Given five minutes, can you turn to the person next to you and describe your research? How about over 15 minutes in front of a class of 10th graders? Thinking of one of your research graduate students, how would you prepare her/him to make such an activity equally beneficial for her/him and the 10th graders? For many of us, these are skills only nurtured through conference talks and time within the profession. The __SF-State (CM)²: San Francisco State University Creating Momentum through Communicating Mathematics program__ worked to change this, creating a program that developed mathematics graduate students who could have this conversation and were better engaged in why they were studying mathematics and what role they wanted to have in the future of our profession. As an NSF GK-12 program, (CM)^{2} ran from 2009-2014, working with master’s level students in mathematics to engage them in mathematical discourse while also supporting their research and professional development. Over the course of the five-year program, a total of 43 SF State mathematics graduate students were involved in the project, spending considerable time and energy on K-12 activities in 13 schools in the greater San Francisco area. A key goal was to strengthen the graduate students’ communication, teaching, outreach, and teamwork skills by immersing them in mathematics classrooms and the San Francisco Math Circle. A second key goal was to make mathematics, especially algebra, and its career connections more relevant and explicit for 6-12th grade teachers and students. This post will share successes, lessons learned, and resources for you as a faculty member to build aspects of outreach, teaching, and professional development programs for your own students.

(CM)^{2} provided funding support for nine Ph.D.-bound graduate students per year, with the understanding that these students (the “Fellows”) would work 10 hours per week in an education-related environment. A lot of work went into preparing these students for that experience, with a year-long schedule starting with bi-weekly summer workshops focused on research and applying for Ph.D. programs, culminating in a two-week intensive training to prepare graduate students for the classroom. We reinforced the lessons from the summer throughout the year with weekly graduate student meetings and larger monthly meetings of graduate students with their partner K-12 teachers. Our goal was to ensure that the graduate students spent the summer focused on making research progress prior to the intensive school-year schedule. The partner teachers attended the second half of the two-week training, meeting their mentor students and starting to create lessons and schedules for the upcoming year.

At the first workshop, (CM)^{2} Fellows were asked to turn and explain their research to the student sitting next to them. Their goal was to prepare a lesson for middle/high school age students to help them learn about the graduate student’s research, and the first step in this process was to describe their work to a colleague at the same academic level. This was a struggle, as everything you might expect happened with the biggest issue being Fellows’ use of precise language specific to their area of research. They were so focused on showing how much they knew about their field that they failed to concentrate on communicating their work in context. When first presented with the exercise, the Fellows struggled, but with practice their explanations were more prepared for a K-12 classroom. From there we scaffolded the progression by having the graduate students explain their work to their mentor teachers, whom they would be spending the majority of their classroom work time with. This mentor team then worked for several workshop days to create a 15-minute sample lesson to share with the group of 9 mentorship teams. With feedback from the other graduate student and teachers, they worked over the next year to create grade-level appropriate mathematics activities that were related to their academic research. Over the year, the Fellows further developed these into 5-10 page lesson plans with an introduction to the mathematical content, an overview of the lesson structure and directions for implementation, worksheets, and a summary of how the lesson went, including recommendations for the future. These mathematics lessons were developed, shared between partnership teams, and papers to accompany them were posted online; you can still see many of these lessons at __http://math.sfsu.edu/cm2/materials.php__. Development of the lesson into such a report furthered the Fellows’ mathematical communications skills with writing practice.

Creating this lesson plan wasn’t the only aspect of the (CM)^{2 }program. Once classes started, the graduate students quickly fell into a routine of weekly commitments, which included 10 hours of GK-12 student interaction (in class and at Math Circle), a one-hour GK-12 seminar, their own coursework and research, and continued work on their Ph.D. applications. Several of the fellows completed NSF graduate research fellowship proposal (GRFP) applications while participating in the fellowship program.

__San Francisco Math Circle (SFMC) __was key to the success of the (CM)^{2 }program, providing the ideal stepping stone for graduate students to get involved with mathematics activities for middle and high school students. The program had been set up so that the graduate students worked in some of the schools we were using for our satellite programs (including Mission High School, Thurgood Marshall Academic High School, Lowell High School, and June Jordan School for Equity). While in the classroom, the graduate students created a mentorship relationship with the younger students whom they encouraged individually to attend Math Circle. The graduate students then served as instructors at the high-school Math Circle programs. We believe this improved the quality of instruction at the high-schools programs because the graduate students had the opportunity to incorporate more of their research into their Math Circle presentations and were better able to moderate their presentations for the correct level of students based on their experience in the classrooms of the students they were working with. In addition, the (CM)^{2 }program sponsored nine teachers who were involved in the Math Circle. Overall, partnering with the (CM)^{2} program helped SFMC address not only the challenges in maintaining the diversity of the students in the Circle, but also that of increasing the number of qualified teachers willing to participate in a Circle with the students.

The dichotomy of our Fellows’ work, including both in-class and out-of-school Math Circle experiences, provided an informed balance that helped them become better mathematicians and teachers. The in-class work provided a model for how to work with students of this age, giving reinforcement to expectations of behavior and any aspects of student discipline. Here are three sample testimonies from Fellows that show the diversity of experiences and highlights this program provided for the participating graduate students:

“I’m serious about becoming an educator. Readings and discussions in 728 were terrific, and I learned more about teaching from working with […] than I did last year teaching on my own. My work at […] and SFMC were more valuable to me than the paychecks..”

“(CM)² has also enhanced my preparation to enter the Ph.D. programs. Being in this program has increased my confidence and excitement to start my Ph.D. program through the support from Matt and Brandy as well as the teachers and other fellows. In addition, the financial support has opened up opportunities by removing financial barriers […] As an underrepresented minority, this program has given me so much in terms of academic support, encouragements, and resources. Through (CM)², I have become more aware of the low numbers of underrepresented minorities in higher education and I’m ecstatic to be a role model to others.”

“In all honesty, I was not interested in working in the high school or in math circle to begin with but obviously took this fellowship for financial and moral support. Now, I’m looking sadly at the end of my time working at […] High and with […] and […] and the other teachers. I’ve learned so much, even if I never teach again, well, I may start up my own math circle someday, but even if I don’t, it has been a very personally rewarding experience.”

The out-of-school work provided creative outlets for the graduate students to discuss their research and higher level mathematics that didn’t need to be connected to curriculum goals or other K-12 benchmarks. We also saw success reflected in comments we received from partner teachers:

“Having another person in the room with deep math knowledge is good for the students and definitely helps with the amount of material covered. As to my pedagogy, I’m sure it has had an effect but I haven’t tried to define what it is. It has caused me to think more about what engages the students. I tend to push moving through the concepts, but now I am thinking more about ways to get the students involved. I don’t believe this necessarily has to be a show of how the concepts are relevant to the students lives. Coming up with intriguing problems where the students don’t feel shut out and turn off because of prior assumptions of them having a particular skill set seems to work well. Just having someone for me to interact with intellectually has been good, and I think it will help in improving course content.”

“Having another pair of knowledgeable eyes in the classroom has pushed me to think about and to be able to articulate what my motives and intentions are for each lesson. […] is always thinking beyond the math concept to its implementation and I have benefited from his sharing his insights. Also, the students love it when we come up with different interpretations on a topic.”

Our GK-12 program provided us with an invaluable opportunity to create a community of scholars, from elementary students to university faculty — vertical integration at its best. Working with our graduate students, their partner teachers, and their K-12 students was both great fun and an interesting challenge; everyone could learn something from the other participants (and that most certainly included us). We are somewhat heartbroken that the NSF cut the overall GK-12 program. The impact of our program went beyond the students; e.g., the San Francisco State mathematics department continues to offer a graduate-level writing-in-the-discipline course, which was initially developed as part of (CM)^{2}. Some of our current graduate students continue to be involved in the San Francisco Math Circle, including teaching and leadership positions. We hope we’ve inspired you to bring an aspect of this or a similar program into your work and encourage you to contact us for further information. Finally, we thank the graduate students and teachers who we worked with for their contributions and support of our program.

Consider how you would respond to two different versions of a question. In the first, you are asked to solve a high school mathematics problem. In the second, some high school students’ solutions to that problem are shown to you. You are asked to assume the role of the students’ teacher and to evaluate the mathematical validity of the students’ different approaches. What knowledge, if any, do you need in the second situation that you don’t need in the first situation?

Some would argue that the second situation is just about knowing math. If you, yourself, can solve the high school mathematics problem correctly, and you are very capable in high school mathematics, then this should be enough to evaluate a high school students’ solution. Yet others might say that this question is about teaching. If you can’t interpret students’ work, you can’t judge it accurately. Still others might say that this question targets something in between straight math and teaching. We would say that this scenario assesses a blend of all of these things that previous scholars have named *mathematical knowledge for teaching *(MKT). We ask the reader to join us in considering, as some have argued, why MKT is a form of applied mathematics – and why mathematicians have a stake in thinking about MKT in this way.

To begin, consider this example:

This question illustrates a coordination of mathematics and teaching. The intended answer is that only Matt’s method is valid. Jing found the slope between two points and assumed the slopes between all points were the same, without verifying that the data were actually linear. Matt’s method, without producing the function, verifies that all the given points fall on the line. The key idea to note is that the assumption of linearity is not warranted by the information given in the problem. Jing makes this assumption, and Matt does not. Solving the question requires knowing that the linearity assumption is not warranted. Solving the question also requires making sense of what Jing’s and Matt’s written work might mean. The linearity assumption jumps out to an experienced teacher as the important point that is likely to trip students up.

The question asks the respondent to think simultaneously about mathematics and teaching in a way that taking conventional mathematics coursework may not prepare a prospective teacher to do.

In a recent validity study (funded by the NSF __[1]__) designed to support a measure of teachers’ MKT, we asked secondary mathematics teachers to respond to this question. We saw variations of all of the above perspectives in their responses. Some focused on the mathematics, others on digging into what the two students might have been thinking, and many on something in between. Notably, respondents who did not attend to the linearity assumption were likely to favor Jing’s response, often describing it as more “complete” or “sophisticated”. In a number of cases, respondents stated that Matt needed to check all three points because linearity was not given, but also indicated they would have given at least partial, if not full, credit to Jing’s answer. After all, Jing derived the equation, whereas Matt only verified it. Yet, from the perspective of strict mathematical correctness, these judgments seemingly hold the two students to different standards. Imagine these respondents bringing this understanding to real students in a real classroom. Students might deduce from this that Matt’s approach is mathematically wrong. They might generalize that only solutions that involve algebraic manipulation of formulas are valid. They might pick up the implicit message that apparent sophistication is more important than whether a method effectively and efficiently addresses a given problem. They might be genuinely confused by the apparently different standards of mathematical validity. And finally, they might take from this that linearity can be assumed when convenient, or simply not have an opportunity to learn that such assumptions need to be established and/or verified as a general practice. In other words, these responses indicated something important: the knowledge these teachers did or did not demonstrate was likely to *matter* in their teaching, for both mathematical and pedagogical reasons.

Scholars have acknowledged the importance of MKT at the elementary level, even mapping out programs of study designed to provide elementary teachers with an opportunity to acquire this knowledge. It has been less clear whether secondary teachers-in-training need to study anything beyond the content of conventional mathematics and pedagogy courses. The study from which the opening question was taken demonstrated that that there *is* MKT at the secondary level that (1) can be measured and that (2) differs from what teachers are likely to learn in conventional mathematics courses. And we suspect that secondary mathematics teachers-in-training have few opportunities to learn this MKT as part of their teacher preparation programs.

While it seems clear that MKT learning opportunities belong in teacher preparation programs, it is less clear whether they belong in mathematics courses, pedagogy courses, or elsewhere. One could argue, as some have done, that MKT is applied mathematics; applied in the context of teaching. We ask the reader to join us in considering why one might think of MKT as a form of applied math in this way and present three arguments for why mathematicians teaching mathematics to teachers-in-training might have a stake in thinking about it this way.

- MKT is, at least in part,
*math*. Although a teaching perspective makes the linearity assumption in the opening example jump out to an experienced teacher, attending to such assumptions is also*mathematically*crucial. Often, mathematical ideas are pedagogically key because they are mathematically key. Because students are still learning how to engage in mathematical thinking, teachers need to do more than just think mathematically themselves; they also need to know how to call attention to what they are thinking and why. This involves understanding the mathematics in more than tacit ways. It is not enough for a teacher to avoid the error of assuming linearity; a teacher needs to know to recognize the linearity assumption as important, pedagogically. But both kinds of understanding are grounded in the mathematical idea that linearity cannot be assumed. - As with other forms of applied math, learning components in isolation may be less useful than learning them in context. Finding solutions to differential equations doesn’t get you far as a mathematical biologist if you haven’t had a lot of practice using them to approximate natural phenomena. In the validity study, we saw striking cases in which respondents with strong mathematics backgrounds and who demonstrated strong, specific, but decontextualized mathematical knowledge in other parts of the interview did not apply that knowledge appropriately to contextualized problems. In other words, we should not assume that studying mathematics in traditional courses necessarily or automatically equips teachers to call on and apply that knowledge appropriately in their work.
- We all have a long term stake in improving teachers’ MKT. Teachers were once students and learned mathematics in school. Their students are future teachers. The authors of
*Mathematical Education of Teachers*2 urge us to remember Felix Klein’s “double discontinuity,” that a teacher-in-training doesn’t see how university and high-school mathematics connect, and upon entering the classroom, doesn’t see what they learned in university has to do with teaching. As part of the second discontinuity, prospective teachers who are able to use quite sophisticated reasoning or sound mathematical habits of mind, such as checking assumptions in advanced math classes, might be capable of applying those skills to high school mathematics that they will be teaching, but simply have never had the occasion to do so.

What might we do to create learning opportunities for preservice teachers to learn MKT as applied math? We recently convened a group of mathematicians and mathematics educators to take on this question. Participants considered questions like the opening one as examples of the MKT we would ideally expect teachers-in-training to have opportunities to learn in teacher preparation. Workshop materials included a set of 55 such questions. (We encourage you to ask us more about these questions! They were designed with the intention of giving more examples of what MKT is like, especially at the secondary level. Copies of these questions can be obtained by contacting Heather).

When we began this workshop, we had hopes that these questions could serve as potential in-class or homework questions in both mathematics and pedagogy courses. The workshop provided some encouragement that our hopes could be realized, and that as a bonus, the questions could seed productive conversations among colleagues. For instance, our participants suggested that questions could help in identifying mathematical ideas that might be missed by both mathematics and pedagogy courses because they do not fall squarely into either. They suggested using the opening example as a context to discuss what constitutes mathematical validity. Our participants further suggested that the questions could be concrete examples for conversations about the tension between the notion that in mathematics, one wants to “never say something you have to take back,” yet in teaching, one should not “say something a student can’t take in;” and that questions can be mini-cases that could develop into extended cases of teaching and mathematics. Some participants are making plans to use these questions in their programs.

Based on the results of our validity study, and the promising suggestions arising from the workshop, we are now turning our attention to supporting the use of these questions to supplement coursework, and to forming professional communities focused on learning and teaching MKT through the use of questions such as the opening example. We invite your thoughts and comments about your experiences teaching MKT as a form of applied mathematics, especially at the secondary level, and would welcome requests to share materials.

[1] This material is based upon work supported by the National Science Foundation under Grant #DGE-1445630/1445551 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

]]>

Anyone who teaches mathematics in the US knows that the quality of education could be better, but we also know that the problems are complicated and defy easy solutions. I grew up in Ontario, Canada, where I attended high school and completed an undergraduate degree in mathematics. Afterwards I completed a Ph.D. in the United States and I have now been teaching undergraduate mathematics here for over ten years. These experiences suggest to me a change that would improve college mathematics education in the US. It won’t solve every problem, but it is something concrete that we can do right now.

Suggestion: Replace the typical one-semester “introduction to linear algebra” course with a two-semester linear algebra sequence. This would be taken in the first year of college, in parallel with calculus. It would not have calculus as a pre-requisite.

In effect, this would place linear algebra and calculus side-by-side as the twin pillars of undergraduate mathematics. I believe this would have several immediate benefits for the curriculum. In this blog post I’ll describe three of these benefits and then I’ll explain how my experience as a student in Canada and as a professor in the US has brought me to this position.

**Three Fundamental Problems**

*Linear algebra is being undersold. *Linear algebra is the common denominator of mathematics. From the most pure to the most applied, if you use mathematics then you *will* use linear algebra. This is also a fairly recent phenomenon, historically speaking. In the 19th century, linear algebra was at the cutting edge of mathematical research. Today it is a universal tool that every user of mathematics needs to know. This becomes more true every year as algorithms and data play a bigger role in our lives. It seems to me that the current curriculum was fossilized in an earlier time when linear algebra wasn’t so useful. But times change; sometimes we need to re-examine the mathematics curriculum to see if it is still relevant. I believe that a two-semester linear algebra sequence in the first year will be a more honest representation of how mathematics is used today.

*Complex Numbers. *Complex numbers are currently an orphan in the undergraduate curriculum. According to the Common Core Standards, students are supposed to learn about complex numbers in high school; however, from my experience with US college students I know that they are not learning this material. At the University of Miami (where I teach), the basic ideas of complex numbers including de Moivre’s Theorem appear in our pre-pre-calculus remedial course, so it is not reasonable that a student who takes this class will have time to complete the math major in four years. Therefore, in practice, we are pretending that our undergraduates have seen complex numbers when this is not the case. It seems that many of our students are introduced to complex numbers for the first time in an upper-level complex analysis course. In my opinion this is way too late. If US students are not to learn about complex numbers in high school, then it seems to me that the first semester of a two-semester linear algebra sequence is an excellent place to introduce this material. Here they can see the 2-dimensional geometric interpretation of complex numbers via 2X2 rotation and dilation matrices. This would have the added benefit of teaching complex numbers to future high school teachers (who are usually not required to take a complex analysis course).

*Linear Algebra remediation eats up other courses. *At the University of Miami we require all math majors to take MTH 210 (Introduction to Linear Algebra). As is typical in many departments, this is a one-semester course that is usually taken in the second year, as it has Calculus II as a pre-requisite. Most of the students have never seen vectors or matrices before, so our goal is to get them from the basic ideas to the useful applications in one semester. This course is then required as a pre-requisite for many upper-division courses. However, most instructors find that the students’ subsequent linear algebra background is very shaky because one semester is not enough to absorb all of the material. In practice this means that many upper-division courses must include a crash course in the relevant ideas from linear algebra, which is hugely inefficient. A two-semester first year sequence would ensure that teachers of upper-division math courses could assume that the students are familiar with the basic ideas of linear algebra. This would save time and allow each of these classes to cover more material.

**This Has Been Done Before**

I attended high school and college in Ontario, Canada between 1993 and 2002. In those days we had 5 years of high school; the fifth year (called OAC for “Ontario Academic Credit”) was optional for the general population, but it was required for entrance to many university programs. The OAC mathematics curriculum consisted of three courses: Algebra and Geometry, Calculus, and Finite Mathematics. A student entering a Bachelor of Science program at an Ontario university was expected to take all three of these. The Calculus course was similar to the AP Calculus course that exists in American high schools today. The Finite Mathematics course was an introduction to probability and statistics with a discussion of induction and elementary combinatorics. The OAC Algebra and Geometry course was a novelty that has no parallel in American schools. (Since the OAC program was phased out in 2003 it no longer exists in Canada either.) To give you an idea of the course, here is the table of contents from the final version of the course textbook (now out of print).

The curriculum was mostly analytic geometry in 2 and 3 dimensions providing a thorough introduction to representing lines and planes in 3D. It also introduced Gaussian elimination, orthogonality, the language of vectors with applications to physics, matrices as linear transformations, and complex numbers. I remember this course very vividly as one of the experiences that led me to become a mathematician. Ontario universities required these OAC courses for entrance to certain programs. The undergraduate curriculum was then able to make good use of these courses. For example, when I entered Queen’s University as a mathematics and physics major I was required to take two full-year mathematics sequences in my first year. One sequence was Calculus and it required OAC Calculus as a pre-requisite. The other sequence was Linear Algebra and it required OAC Algebra and Geometry as a pre-requisite. Let me emphasize this situation:

I was required to take a semester of linear algebra in high school, which was then followed by a required two-semester linear algebra sequence in the first year of college.

The textbook that we used for Linear Algebra was the 3rd edition of “Elementary Linear Algebra” by Larson and Edwards. This text is indistinguishable from the standard textbooks used today for the “introduction to linear algebra” course in the United States. Thus, by this point I had seen the same material, but I had seen it over three semesters instead of one.

After the first year sequence was an additional one-semester course in linear algebra that was required for mathematics majors with a concentration in statistics or physics. (It was also required for many engineering programs.) This course gave a more abstract introduction to inner-product spaces, with applications to function spaces. The students in the course were diverse, coming from many different departments. This indicates to me that the material of this course was still seen as universal in the sense that it was relevant to all users of linear algebra, whether pure or applied. It was not a special topics course.

In summary, by the time I had completed a Bachelor of Science degree in mathematics with a physics minor, I had been required to take *four semesters of general-purpose linear algebra*. And you want to know something funny? As I proceeded to graduate school in pure mathematics at Cornell, I shortly came to feel that the most serious deficiency of my undergraduate education was that *I had not seen enough linear algebra!* Not only did I find my multilinear algebra background weak when I learned representation theory, I was also shocked when I learned about the Perron-Frobenius theorem and its amazing applications (e.g., to ranking webpages): Why had no one told me about this theorem?

**My Teaching Experience**

I joined the mathematics faculty at the University of Miami in 2009, and since then I have taught our Introduction to Linear Algebra course (MTH 210) four times. After this, many upper-division courses (such as Multivariable Calculus) list MTH 210 as a pre-requisite. We have two upper-division linear algebra courses in the curriculum (one more abstract and one more numerical), but these are regarded more as capstone courses and they do not serve as pre-requisites for anything else. Thus MTH 210 is a very important course.

Let me describe my experience teaching the course and how my approach to the syllabus has evolved. The official department syllabus reads as follows:

Vectors, matrix algebra, systems of linear equations, and related geometry in Euclidean spaces. Fundamentals of vector spaces, linear transformations, determinants, eigenvalues, and eigenspaces.

The majority of students in the course have never seen the language of vectors and matrices before, nor have they taken a physics course which could be used for motivation. In my opinion, it is completely impossible to cover the material from a standard textbook under these conditions, so my experience with the course has been a painful process of deciding what to leave out. After trial and error I have decided that the abstract notions of “vector space” and “linear transformation” had to go, so that I can get to some substantial applications by the end of the semester. The language of matrix arithmetic is already plenty abstract for students seeing it for the first time.

Since the students are starting from scratch, I begin the course with a thorough introduction to Cartesian coordinates and analytic geometry in 2 and 3 dimensions. Without the visual intuition that this provides I don’t think they’ll get very far. By the halfway point of the semester I want to cover the standard material on Gaussian elimination and to be able to explain the geometric intuition behind it.

After covering these minimum pre-requisites in the first half of the semester, in the second half we dive into the actual “linear algebra”. Here is where the painful cuts come. Since this is the only linear algebra course that most of these students will ever see, I need to cover the significant applications. But I also need to cover enough theory that these applications can be understood. With practice I’ve come up with a way to do this that I’m reasonably happy with. If you want to see the details, my lecture notes from two previous iterations of the course are available here and here. But on the other hand, I’m not happy at all. I know how useful linear algebra is, and I know that most of these students would benefit from a deep understanding of it. I also know that this one-semester class goes by too quickly for the ideas to really sink in. I can only hope that I’ve given them a roadmap so they can fill in the details later when needed.

**Conclusion**

It seems clear to me that a two-semester linear algebra sequence would vastly improve the undergraduate mathematics curriculum in the United States. This would give extra time to absorb the important ideas and give space to discuss neglected topics such as complex numbers and analytic geometry. It would also allow upper-division courses to use linear algebra in a deeper way.

This is natural from a mathematical point of view, but I also recognize that from an administrative point of view it’s pretty radical. I’ve sat on committees and I know how time consuming it can be just to change the title of an existing course. Adding a new course at the center of the curriculum could easily be a decade-long process, and it would be far from painless. Being realistic, I suspect that a change like this will only follow from a discussion at the national level. The MAA’s Committee on Undergraduate Programs in Mathematics has already made some recommendations for the role of linear algebra in the curriculum in their 2015 Curriculum Guide.

I generally agree with the CUPM’s remarks, but I think they are too shy in their recommendations. For example, the committee’s Content Recommendation 1 from the Overview says that “Mathematical sciences major programs should include concepts and methods from calculus and linear algebra.” This language seems to put calculus and linear algebra on equal footing (I assume they are listed in alphabetical order), but it doesn’t grapple with the reality that linear algebra is currently under-represented with respect to calculus. I would rephrase their recommendation in stronger terms:

Mathematical sciences major programs should include concepts and methods from calculus and linear algebra.

These two subjects should be introduced in parallel, and both should be studied in the first year.

Phrasing it this way emphasizes that a change is needed. Undergraduate programs in the US do not introduce linear algebra early enough and they do not place it beside calculus at the center of the curriculum. Other countries (such as Canada) have done a better job with this, and I hope that we in the US can learn from their example.

]]>It could be the punchline of a joke that at any given college or university, at some point, the administration will lean on departments to be more “efficient” by teaching classes in larger sections, or online, or with some technology or another. By the metric of student credit hour to faculty work hour, of course, large lectures are tremendously efficient, and scale admirably. One may argue that there is little difference between an instructor lecturing to 100 or to 200 students, and little difference between an instructor rendered small by the distance to the front of a large lecture hall and one rendered small in the pixels of a video screen. This is the Massive Open Online Course (MOOC) model, which extends this efficiency of scale from 200 to 20,000. Anecdotally, the MOOC tide seems to be receding, but the pressures that argue for this efficiency are not going away. Many departments are being asked to teach, with fewer resources and greater accountability, more students whose mathematical preparation is weaker than in the past [2].

The difficulty here is that the student credit hour metric is easy to measure, while student learning is not. Research says that our efficient passive lecture does not result in the student learning gains we can see with more active teaching techniques [6,7,8]. Indeed, through the Conference Board of the Mathematical Sciences, the presidents of fifteen professional societies in the mathematical sciences have recognized this conclusion and endorsed the use of active teaching methods [4]. But neither the research nor the endorsement provide us with a simple, usable measure by which to demonstrate the effectiveness of these techniques. So our endeavor of teaching remains by easily applied metrics an inefficient one, and I increasingly think one that is inevitably inefficient by even more measures.

This is probably true for all education, but there is a case to be made that it is particularly true for mathematics. For over twenty years, we have watched an accelerating rush to calculus in American high schools [3], and I recall a discussion with Project NExT [9] Fellows in about 1995 in which the observation was made that as calculus is pushed down into the high school curriculum, algebra is pushed up into college. But it feels worse than a one-to-one exchange, because while calculus is, more-or-less, well-defined―though this characterization deserves further evaluation―what bubbles up is not.

As a result, our efficient lecture, in which all students are assumed equally well-prepared and equally well-served by a uniform delivery method, becomes even more poorly suited to reach our students. We are stuck, again, with our most effective―and perhaps only effective―remediation being inherently inefficient: we need the instruction for these students to be individually responsive, not broadly scalable. Further, mathematics is not a field in which students’ understanding is built up from only a small number of physical laws. Thus, the responsive diagnosis and remediation must be nimble and able to evaluate individual students daily, as we navigate in class a varying mathematical terrain that requires varying prerequisite knowledge. As a discipline, the thinking and logic we demand of all practitioners, students included, is that of a science, but in the interconnected myriad details our subject may be more akin to the humanities. (The need for languages to be taught in small sections is rarely questioned; perhaps we may argue that the mathematics education research is really demanding the same for mathematics.)

So to be effective instructors, especially in our present environment, we may claim that we are inevitably inefficient. But I think that this premise extends beyond the classroom, infiltrating even the systemic support that effective instruction requires.

I wrote above that calculus is, more-or-less, well-defined, and I think this is true. Comparing two arguably different calculus textbooks―Stewart’s Calculus and the text of the same name by Hughes Hallett, et al.―reveals 28 sections that cover essentially the same material and only four that are demonstrably different in mathematical content. But calculus courses themselves are, by dint of institutional constraints (student preparation and needs foremost among them) far less uniform, and this difference in courses between institutions gets only more pronounced as we move on from calculus. To some degree this has limited import when students stay at one institution through graduation, but this is becoming less and less the case. The push, and need, for greater affordability of higher education means that increasing numbers of students may be transferring between colleges (especially two-year colleges) and universities. Thus there are increasing numbers of students who have taken courses at other institutions entering our classrooms, and if they have unexpected gaps in background knowledge we need to give the instructor time and contact to be able to diagnose and remediate that. For this to be at all a feasible undertaking, we need first to ensure that students are in the right course in the first place, which requires a “diagnosis” of any courses with which our students arrived. I think that this diagnosis is also one that is necessarily inefficient. It is not one that someone without knowledge of the subject can do reliably, and thus for it to be done well we must use expert faculty time to do it. This use of faculty time is also not efficient by any standard business model: we are using our most highly trained employees in the organization to evaluate on a student-by-student basis these course requests.

I won’t argue that this is the only way to evaluate transfer credit, but I think it is an effective way to do it, even as it is by some metrics inefficient. I’ve evaluated perhaps 200 courses for equivalency to courses at the University of Michigan in the past year. And while this doesn’t show up in the list of teaching duties that I have performed in that time, I believe it to be a service that allows our teaching to be effective.

I watched a similar, and similarly inefficient, evaluation unfold this summer in the course of several meetings of the faculty who are most directly involved in the administration of our Introductory Program (loosely, our course preceding calculus, calculus I, and calculus II). The University has a summer program to promote diversity in STEM subjects, and asked if we could designate some sections of calculus I for those students so that they could enroll in calculus with other members of their cohort, and in sections taught by instructors the students already know. From the perspective of this program these are obviously desirable outcomes. However, it also has the potential to isolate students in those sections who are not part of the cohort, and requires that the Introductory Program Directors establish these sections in advance so that this is possible. These latter outcomes are less desirable, and the former is a significant concern for the learning of other students in those sections. As a result, this evaluation was not a straightforward one, and it unfolded over the course of two or three meetings of the five to seven people involved in these decisions. There are perhaps 18 students involved in this program for the fall. To me this seems to fall in the category of inefficient processes, at least as measured by the time spent by the decision makers.

At any given college or university that is trying to do a good job in teaching mathematics I suspect there will be similarly inefficient systems supporting the inefficient work of the mathematics teaching itself. They may not―arguably, will not―be doing the tasks I’ve picked here. Because of the nearly infinite variation in the colleges and universities across the country and world, the systemic challenges at each will be correspondingly different and varied. But because the difficulties in dealing with differently prepared students in all of these environments is the one thing we are certain will be constant, it’s hard to imagine the systemic issues will ever be absent.

It is said of at least some theoretical mathematicians that they are proud that their chosen studies do not have (visible) practical applications. I think that we as mathematicians who are concerned with the effectiveness of our institutions at educating students in our chosen field should perhaps be similarly proud of our inefficiency by practical measures. Active learning really is better, and is better done on a scale at which there is significant student-faculty interaction [5]. And our students learn best when the systemic support for these active learning classrooms allows them to operate at their best. Insofar as these are inefficient, these inefficiencies are inevitable. Thus they are also an essential characteristic of the effective teaching of mathematics.

**References**

[1] ALEKS. (2016). Accessed Aug. 31, 2016.

[2] Bressoud, D. (2015) Calculus at Crisis I: The Pressures. Launchings. (May 1, 2015). Accessed Jul. 7, 2016.

[3] Bressoud, D. (2015) Calculus at Crisis II: The Rush to Calculus. Launchings. (Jun. 1, 2015). Accessed 16 May 2016.

[4] CBMS Statement on Active Learning in Post-Secondary Mathematics Education.

(July 15, 2016). Accessed 24 Aug 2016.

[5] Chickering, A.W. and Z.F. Gamson. Seven Principles for Good Practice in Undergraduate Education, New Directions for Teaching and Learning #47. Jossey-Bass (1991).

[6] Freeman, S., et al. (2014). Active Learning Increases Student Performance in Science, Engineering and Mathematics. Proceedings of the National Academies of Sciences, 111(23):8410–8415.

[7] Kogan, M. and S.L. Laursen (2014). Assessing Long-term Effects of Inquiry-based Learning: A Case Study from College Mathematics. Innovative Higher Education, 39(3):183-199.

[8] Laursen, S.L., M.L. Hassi, M. Kogan and T. Weston (2014). Benefits for Women and Men of Inquiry-based Learning in College Mathematics: A Multi-institutional Study. Journal for Research in Mathematics Education, 45(4):406-418.

[9] Project NExT. Accessed 24 Aug 2016.

This is an announcement to the mathematical community that the White House Office of Science and Technology Policy (OSTP) has issued a Call to Action for incorporating Active Learning in K-12 and higher education STEM courses. Their call includes a submission form where they ask: “What new (i.e., not yet public) activities or actions is your organization undertaking to respond to the Call to Action to improve STEM teaching and learning through the use of active learning strategies?” The deadline for submission of responses is Sept 23, 2016. It would be excellent for the mathematics community to be well-represented among the submissions, so I encourage our readers to submit their activities and to share this information with others.

]]>*Editor’s Note: An expanded version of this article previously appeared at **http://openpyviv.com/2016/07/12/ECCO/**.*

Being one of the few women in the men’s world of mathematics and computer science has led me to look around and spot our flaws when inclusivity is concerned. Let’s not fool ourselves: even though we think of us as being purely objective beyond bias, the maths world is not an inclusive paradise. The academic world I personally live in is made of mostly white men, mostly from western countries (Europe, US, Canada, and just a little bit of Asia). If you look even closer, you’ll see that most of us come from well-off educated families. Except for the fact that I am a woman, I check all the other boxes myself and I am well-aware of it. Considering the multiple causes of this situation, what can be done? What can I do as a single individual in this world, when I’m busy fighting my own fights earning my right to stay around? Well, I’m not going to answer that just now, but I will share a very good experience I just had. I went to a CIMPA maths summer school in Colombia that was different: ECCO 2016. For the first time, I felt it was indeed inclusive in the best possible way. And, it was excellent maths too, so I was really happy.

First, a little bit of context. As opposed to a classical conference where most presentations are short ones to announce new results, a summer school is usually made of mini-courses on a certain topic. At ECCO 2016, the main audience was made of students (masters students, PhD students, and undergrads) but some postdocs and even professors participated as well, as we are always keen on learning new things. It was in Colombia and the topic was combinatorics, which happens to be my field. ECCO runs every two years, and began in 2003 as a small event organized by Federico Ardila. He is a Colombian mathematician based in the US and we (the academic world) owe him thanks for many great researchers in combinatorics. I had noticed before that the number of Colombian people among researchers in combinatorics was astonishingly high, but before I met Federico I had no idea why. Most of this very active Colombian community is now organizing the conference. Over the years, ECCO has become quite a big event in combinatorics with a very positive, well-earned, reputation. This year, for the first time, it was a CIMPA school and there were over 100 participants. So why was this conference so good?

**Background diversity. **One thing that I found surprising is that the students came from very different knowledge backgrounds: some of them were undergrads, some were Masters degree students, some were PhD students, some had experience with combinatorics, some did not. And of course, there were also postdocs and professors as I mentioned. Honestly, I didn’t think it was possible to make a conference that was interesting for so many different people with such a variety of knowledge bases. And still, they did it. I think the main reason for this was they intended, from the beginning, that their conference be accessible and interesting for the entire audience instead of just a narrow selection. I am pretty sure they gave detailed instructions to the teachers. The classes themselves were high level mathematics, as you would expect from any summer school. So, of course, not everyone understood everything (that never happens): you cannot expect an undergrad to perfectly follow a condensed high level course on a subject he/she has never heard of. But it was done in a way that everyone could get something out of it. I learned very interesting subjects which gave me new ideas for my research, and undergrads could get direct insight of what combinatorics was about, often understanding much more than I would have expected.

**Country diversity. **That was probably one of the nicest aspect of the conference. Being in academia and travelling a lot, I get to meet people from a bunch of countries, but I don’t think I had ever seen that many nationalities! Of course Colombians and other South Americans, but also North Americans, and Europeans, and more. I counted 23 nationalities, most of them students. Academia is a lot about networking, but it is a very difficult network to enter when you come from the wrong country, so such events can really change the way things are. Also, I liked that it broke the old colonialist structure of “western teachers” spreading the knowledge to poor students from “left out” countries. It was an international crowd listening to teachers from an international background. It was European and North American students coming to Colombia to get maths knowledge along with the Colombian students.

**Women. **Let’s stick to numbers: I counted about 25% of women among participants, and two classes out of four were taught by women. Believe me, these are quite good numbers. On the first day, all speakers were women and I’m not even sure it was intended!

**Code of conduct.** The first time I heard about the notion of a Code of Conduct was when I started attending programming conferences, especially PyCon. The very idea appeared quite odd to me. To a French person, the idea of a list of rules often strikes as prudery, especially when it comes from America. It is also very far from the spirit of maths conferences where the idea is, basically, that you only care about the maths stuff and the rest is mostly irrelevant. I do believe we should find a way to bring the idea of the code of conduct to the maths world but I have no idea how. It looks like such an exhausting lost cause and I have no time or energy for it. And so, I was very surprised to see that ECCO had a community agreement which was basically the same thing! I thought it was well-written, emphasising the diversity of the conference and the way to make it a comfortable place for everyone. I believe it was well enforced, though I cannot know first hand. But what I could see is that the organizers made some time for us to read it and also, later in the week, to come back to it and discuss it. My academic colleagues were a bit taken aback as they had never even heard of such a thing. But I will conclude this paragraph with a quote from a female participant:

I was first very surprised and looked at it as an oddity. Then I remembered what it was being a grad student at conferences and of all the weird guys I had to avoid. So I figured, yeah, why not.

**Language. **The conference was in English, as it is the common language in academia nowadays. But a special effort was made towards Spanish speakers, especially Colombian students so that they wouldn’t feel left out by the language. All the announcements were made in both languages. All the class material was translated in Spanish, either by the speakers or the organizers. Also, I felt that the mathematical language was made accessible all throughout the conference by using clear simple definitions without prerequired knowledge.

**Exercise sessions. **It is typical to have exercise sessions in a summer school, but here they were special. The organizers had a very simple and great idea: each day, we would get a random number that would determine the 3 to 5 person group we would be working with. Sometimes, the randomness was a bit modified to allow a uniform distribution of the professors among the groups. The result was great: I got to work with different people everyday, it gave me a good reason to participate and work on those exercises, I got to meet most of the students, I felt useful as I could use my knowledge to help the students understand the course material. It was a great time for the students, especially undergrads, to review the class material in a casual atmosphere where they had people around to answer their questions. Also, the exercises themselves were really well thought: they included very basic questions so that everyone could familiarize themselves with the class content but also advanced problems for those who already knew a bit and could go faster. At the end of the session, some students would go on the board and do the exercises. It was a good occasion for everyone (undergrads, master students and beyond) to show what they had understood, to make them confident that they were able to solve problems even though the course looked hard and they might have not understood a word when they first heard it.

**Questions. **This is a little detail but one that quite summarizes the spirit of this conference. After a few days, the organizers noticed that questions were coming mostly from the most experienced participants (postdocs, professors). It is indeed very hard to ask a question after a talk: you have to speak up in front of everybody, you feel like you didn’t understand much, that your question is just going to sound stupid, that you will sound stupid in front of everyone… So at some point, the organizers decided that the first question after each talk should come from an undergrad or a master student. It meant we professors had to wait a bit until one of them would feel strong enough (and pushed by the awkward silence) to speak up. They were not stupid questions! I am not sure we stuck to this rule up to the end but it definitely helped “break the ice” for the younger students and make them feel like their questions were welcomed.

**Panel. **As I said, in most maths conferences, everything that is not maths content is often thought as irrelevant. It was not the case here. Proof is they organized a panel where participants could ask questions to people at different points in their career: an undergrad, a grad student, a postdoc and a professor. I wasn’t there myself (I was visiting the great city of Medellin as you can read here in French) but I think it is a great idea. Most students have no idea what it’s like to work as an academic; they are entering the unknown. For many Colombian students, it often means applying to foreign programs when they have never left their home country. Getting a little feedback from people who are already out there is quite helpful!

**Reaching to the outside world.** There was a unique effort to connect the conference with the *outside world*: a high-quality and successful public lecture was given by Federico Ardila during the time of the conference and some organizers took part in a program for high schoolers. It was a great pleasure for me to see so many people being curious about mathematics, about knowledge. And I really liked the fact that it was connected to the summer school.

**Conclusion. **It worked. It was a great event! I believe everybody left with the feeling that they learned a lot and lived a great experience. Most people were staying in the same hotel and we would go out together, having dinner, going dancing (salsa!). At 3AM on the last day, after a great evening of salsa dancing, the students would not leave the bar that was trying to close down. They would say goodbye for ever, exchanging vows to stay in touch like young teenagers after a summer camp.

Not all conferences are like this. Actually, none of them are. I can understand that every conference has a different purpose, we cannot just apply everything everywhere. But, we could take this event as an example: I know I will. For me, this is how maths should be most of the time, not two weeks every other year.

]]>To start, I want to thank all of our readers, subscribers, and contributors — we appreciate your feedback and ideas through your writing, social media comments, and in-person conversations at mathematical meetings and events. Since launching our blog in June of 2014, our articles have received over 189,000 unique page views! We will continue to strive to provide high-quality articles on a broad range of topics related to post-secondary mathematics, and we welcome your feedback and suggestions.

I have a few changes to announce regarding the editorial board for *On Teaching and Learning Mathematics*. Following two years of service as a founding Contributing Editor for our blog, Elise Lockwood is leaving our board to join the editorial board of the new International Journal on Research in Undergraduate Mathematics Education (IJRUME). Many thanks to Elise for her excellent contributions that helped the blog have a great start over the past two years! I know that we can look forward to hearing more from Elise in the future as a contributing author.

For 2016-2017, I am happy to welcome three new Contributing Editors to the board:

- Luis David García Puente, Sam Houston State University
- Jess Ellis, Colorado State University
- Steven Klee, Seattle University

Luis, Jess, and Steve bring with them a wealth of expertise in teaching, research, and mentoring, and I am excited that they will be sharing their expertise with our readers.

For those of you who are regular readers, we will continue to publish articles roughly every two weeks, with a target goal of publishing 24 articles per year. Our next post is scheduled for August 22, 2016, where we will hear from Viviane Pons about her experience at a math summer school that was “inclusive in the best possible way.” Stay tuned!

]]>(Note: Authors are listed alphabetically; all authors contributed equally to the preparation of this blog entry.)

Concept inventories have emerged over the past two decades as one way to measure conceptual understanding in STEM disciplines, with the Calculus Concept Inventory (CCI), developed by Epstein and colleagues (Epstein, 2007, 2013), being one of the primary instruments developed in the area of differential calculus. The CCI is a criterion-referenced instrument, measuring classroom normalized gains, which specifically is the change in the class average divided by the possible change in the class average. Its goal was to evaluate the impact of teaching techniques on conceptual learning of differential calculus.

While the CCI represents a good start toward measuring calculus understanding, recent studies point out some significant issues with the instrument. This is concerning, given that there seems to be an increased use of the instrument in formal and informal studies and assessment. For example, in a recent special issue of PRIMUS (Maxson & Szaniszlo, 2015a, 2015b) related to flipped classrooms in mathematics, three of the five papers dealing with calculus cited and used the CCI. In this blog we provide an overview of concept inventories, discuss the CCI, outline some problems we found, and suggest future needs for high-quality conceptual measures of calculus understanding.

Before proceeding, however, we would like to acknowledge and thank the designers of the CCI for starting the process of developing a measure for students’ understanding of calculus. We regret that with the passing of Jerome Epstein, he is unable to respond directly to our findings or contribute to future work. His efforts, and those of his collaborators, have undoubtedly had tremendous impact on the awareness of the mathematics community of concept inventories and the associated need to teach and learn conceptually, and we believe they contributed positively to the teaching and learning of calculus. We hope that the mathematical community will continue the work that he started.

The first concept inventory to make a significant impact in the undergraduate education community was the Force Concept Inventory (FCI), written by Hestenes, Wells, and Swackhamer (1992). Despite the fact that most physics professors deemed the questions on the inventory “too trivial to be informative” (Hestenes et al., 1992, p. 2), students did poorly on the test and, in both high-school and university physics classes, only made modest gains. Of the 1,500 high-school students and over 500 university students who took the test, high school students were learning 20%-23% of the previously unknown concepts, and college students at most 32% (Hestenes et al., 1992, p. 6). Through a well-documented process of development and refinement, the test has become an accepted and widely used tool in the physics community, and has led to changes in the way introductory physics has been taught (e.g., Hake, 1998; Mazur, 1997). The FCI paved the way for the broad application of analyzing student conceptual understanding of the basic ideas in various STEM disciplines (Hake, 1998, 2007; Hestenes et al., 1992), including physics, chemistry, astronomy, biology, and geoscience.

Recently, the authors of this blog post conducted a thorough analysis of the CCI (Gleason et.al., 2015a, 2015b), with the primary objective to assess the degree to which the CCI conforms to certain standards for psychometric properties, including content validity, internal structure validity, and internal reliability. One can think of validity as determining whether an instrument measures what it is intended to measure, and of reliability as determining how well the instrument measures whatever it is measuring. Thus for educational instruments, validity addresses whether a person’s score on an instrument is meaningful with regards to measuring the desired constructs and helps researchers make inferences. The goal of establishing content validity is to determine the degree to which the instrument measures what it intends, and internal structure validity investigates “beneath” the item responses of participants, including how subscales may relate to each other.

Subscales are usually created when there is a desire to understand different components of a knowledge state of the individual or group, and when there is an expectation of different levels of knowledge in the different categories. For example, a high school geometry test might consist of subscales measuring student understanding of triangles, circles, and arcs. Another example would be the different components of an ACT or SAT test where scores are given for English, Math and Reading, as well as a composite score. The items within the subscales should be highly correlated and items between the different subscales may also be correlated, though likely to a much lesser extent. The goal of a validity study regarding the internal structure of an instrument is to determine if the items are measuring distinct constructs or just a single underlying construct in order to justify the usage of subscores.

Since one of the goals of concept inventories is to measure conceptual understanding before exposure to the content, students are required to use their prior knowledge to respond to assessment items at initial enrollment in the course. After completing the course, the concept inventory can then be used to measure gains in conceptual understanding. To ensure validity in this process of measuring gains, items must be carefully written to avoid using terminology taught in the course to which students have no prior exposure.

However, several researchers noticed that released CCI items contained terminology and notation introduced only in a calculus course, such as the word “derivative” and the notation \(f’(x)\). This is problematic because the CCI is meant to only assess students understanding of concepts in calculus, rather than the specific vocabulary of calculus. While a majority (67%) of calculus students at Ph.D. granting institutions have had previous exposure to calculus, 41% of all post-secondary calculus students did not take calculus in high school and high school calculus students have had no previous exposure (Bressoud, Mesa, & Rasmussen, 2015). Items that contain unfamiliar terminology and notation would confuse students and generate responses around random chance for those items. However, Epstein claims they intended the instrument to measures above random chance at the pre-test and to avoid “confusing wording” (Epstein, 2013, p. 7). Though he may have meant “confusing” to mean “convoluted”, a student with no background in calculus would be in a poor position to answer a question in which the notation \(f’(x)\) appears. Because of the use of calculus specific terminology, the validity of pre-test scores is questionable for populations with large numbers of students lacking previous exposure to calculus, such as those in high schools, at community colleges and at regional universities.

With regard to internal structure validity, issues emerged with the CCI when conducting a factor analysis. (For technical details, see p. 1291-1297 of these proceedings.) A factor analysis explores relationships among the underlying factors of the assessment instrument and the cause of those relationships through the analysis of student responses in order to determine the number of underlying factors of the instrument, and their relationships. Epstein and colleagues suggested that the CCI measures conceptual understanding of calculus through three factors (functions, derivatives, and limits/ratios/the continuum). In other words, they claim that a three-factor model captures all of the correlations among the items on the instrument. However, we showed that the item responses are so closely correlated that the total CCI score is explained by one factor, which appears to be an overall knowledge of calculus content, that “can adequately account for the pattern of correlations among the items” and that there are no sub-scales (DeVillis, 2003, p. 109). This finding means the CCI cannot generate valuable information about conceptual understanding of different components of calculus, such as limits or rates of change, but instead is measuring an overall calculus knowledge.

One method of measuring the reliability of an instrument is to measure the extent to which the individual items of an instrument fall within the same general construct. In this regard, Epstein (2013) reported, and the authors of this blog confirmed, that the CCI has an internal reliability Cronbach alpha of around 0.7, meaning that the instrument has a 51% error variance and a standard error of 10% on each individual student score (Cohen & Swerdlik, 2010, Tavakol & Dennick 2011). In particular, this does not meet the established standard of having an alpha of 0.80 or higher necessary for use in research for any type of educational assessment (Lance, Butts, & Michels, 2006, p. 206).

**Conclusion**

With the centrality of calculus to undergraduate mathematics programs and a variety of mathematically intensive partner disciplines, such as economics, physics, and engineering, there is a need to look at the course’s learning outcomes. Recent efforts through the MAA’s National Studies of College Calculus have helped the mathematical community better understand the current state of calculus programs around the country. Data and research on student outcomes in calculus, especially with regards to conceptual knowledge, lag somewhat behind. Part of this is attributable to a lack of appropriate, well-validated instruments to measure outcomes. As most faculty are not trained in rigorous assessment development, they often depend on others for instruments to measure student learning in courses and programs.

Because of the aforementioned concerns, though, we conclude that the existing CCI does not conform to accepted standards for educational testing (American Educational Research Association, 2014; DeVellis, 2012). As such, users of the CCI should be very aware of its limitations. In particular, it may underestimate the conceptual understanding at the beginning of a calculus course for students who have never taken a calculus class before but understand the ideas underlying calculus. We recommend careful consideration in using the CCI and urge users to keep in mind the kind of information being sought. In addition, we suggest exercising extreme caution in using it for any type of formal assessment processes.

Given the shortcomings of the CCI, as well as the inherent limitations of a static instrument with set questions, we argue that there is a need to create an item bank, consisting of rigorously-developed and validated questions on which we have solid psychometric properties, that measure students’ conceptual understanding of differential calculus. Such an item bank would significantly impact teaching and learning during the first two years for undergraduate STEM. Such an item bank could be used by instructors for formative and summative assessment during their calculus courses to improve student learning. The resources could also be used by researchers and evaluators to measure growth of student conceptual understanding during a first semester calculus course to compare gains of students in classrooms implementing differing instructional techniques.

If permission were granted for the CCI to be used as a launching point, then perhaps some of those questions could be used or modified. Prior work on developing conceptually-focused instruments in mathematics, such as the Precalculus Concept Assessment (Carlson, Oehrtman, & Engelke, 2010) and the Calculus Concept Readiness Instrument (Carlson, Madison, & West, 2010), could serve as models for the item-development process.

**References**

American Educational Research Association., American Psychological Association., National Council on Measurement in Education., & Joint Committee on Standards for Educational and Psychological Testing (U.S.). (2014). *Standards for educational and psychological testing*. Washington, DC: Author.

Bressoud, D., Mesa, V., Rasmussen, C. (2015). *Insights and recommendations from the MAA national study of college calculus*. MAA Press.

Carlson, M., Madison, B., & West, R. (2010). The Calculus Concept Readiness (CCR) Instrument: Assessing student readiness for calculus. arXiv preprint. *arXiv*, *1010.2719*.

Carlson, M., Oehrtman, M., & Engelke, N. (2010). The precalculus concept assessment: A tool for assessing students’ reasoning abilities and understandings. *Cognition and Instruction, 28*(2), 113-145.

Cohen, R. & Swerdlik, M. (2010). *Psychological testing and assessment*. Burr Ridge, IL: McGraw-Hill.

DeVellis, R.F. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: SAGE Publications

DeVellis, R.F. (2012). Scale Development: Theory and applications (3rd ed.). Thousand Oaks, CA: SAGE Publications.

Epstein, J. (2007). Development and validation of the Calculus Concept Inventory. In *Proceedings of the Ninth International Conference on Mathematics Education in a Global Community* (pp. 165–170).

Epstein, J. (2013). The calculus concept inventory – Measurement of the effect of teaching methodology in mathematics. *Notices of the American Mathematical Society, 60*(8), 2-10.

Gleason, J., Thomas, M., Bagley, S., Rice, L., White, D., and Clements, N. (2015a) Analyzing the Calculus Concept Inventory: Content Validity, Internal Structure Validity, and Reliability Analysis, *Proceedings of the 37**th** International Conference of the North American Chapter of the Psychology of Mathematics Education, *1291-1297.

Gleason, J., White, D., Thomas, M., Bagley, S., and Rice, L. (2015b) The Calculus Concept Inventory: A Psychometric Analysis and Framework for a New Instrument, *Proceedings of the 18**th** Annual Conference on Research in Undergraduate Mathematics Education, *135-149.

Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. *American Journal of Physics, 66*, 64-74.

Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. *The Physics Teacher*, *30*(3), 141–158. doi:10.1119/1.2343497

Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say?. *Organizational Research Methods*, 9(2), 202-220.

Maxson, K. & Szaniszlo, Z. (Ed.). (2015a). Special Issue on the Flipped Classroom: Reflections on Implementation [Special Issue]. *PRIMUS,* 25(8).

Maxson, K. & Szaniszlo, Z. (Ed.). (2015b). Special Issue: Special Issue on the Flipped Classroom: Effectiveness as an Instructional Model [Special Issue]. *PRIMUS,* 25(9-10).

Mazur, E. (1997). *Peer instruction: A user’s manual*. Upper Saddle River, NJ: Prentice Hall.

Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. *International Journal of Medical Education*, *2*, 53–55. http://doi.org/10.5116/ijme.4dfb.8dfd

Another year has flown by, and so it is once again a good time to collect and reflect on all the articles we have been able to share with you since our last annual review. I enjoyed the chance to re-read all the articles, and I was also surprised at the interesting variety of themes that emerged when I sorted them out. It was not easy to put each article in a unique box, and I will point out the blurring between categories. I hope you enjoy the chance to revisit these articles, and perhaps find new meaning from the juxtapositions here.

**Active learning. ** We devoted two months in the fall to our six-part series on active learning. Taking the article on this subject by Freeman et al. that had recently appeared in the Proceedings of the National Academy of the Sciences as jumping off point, we explored different aspects of active learning. It was exhausting and exhilarating for us to work together as an editorial board to write those articles, starting each new one before all the previous ones were done, and finding new things to say in reaction to ideas that emerged from earlier articles.

- Active Learning in Mathematics, Part I: The Challenge of Defining Active Learning
- Active Learning in Mathematics, Part II: Levels of Cognitive Demand
- Active Learning in Mathematics, Part III: Teaching Techniques and Environments
- Active Learning in Mathematics, Part IV: Personal Reflections
- Active Learning in Mathematics, Part V: The Role of “Telling” in Active Learning
- Active Learning in Mathematics, Part VI: Mathematicians’ Training as Teachers

**Teaching practices.** It should be no surprise that, once again, the bulk of our articles land in this category. Each one discusses something someone has done in their classroom and/or that you can do in yours. But there were some interesting sub-themes that showed up.

**Conceptual, procedural, and modeling:**Whether looking at a framework for integrating the procedural and conceptual, or using modeling, derivative machines, or even our own bodies, all of these articles explore how to include the concrete with the abstract.- Karen Keene and Nicholas Fortune, A Framework for Integrating Conceptual and Procedural Understanding in the First Two Years of Undergraduate Mathematics
- Tevian Dray, Thick Derivatives
- Brian Winkel, Learning Mathematics in Context with Modeling and Technology
- Hortensia Soto-Johnson, Learning Mathematics through Embodied Activities

**High-impact practices:**Maria Mercedes Franco’s article included many high-impact practices she uses, and then Priscilla Bremser focused on one of these practices, service learning.- Maria Mercedes Franco, Why High-Impact Educational Practices (Despite Being So Labor-Intensive) Keep Me Coming For More
- Priscilla Bremser, A Skeptic’s Guide to Service Learning in Mathematics

**Class frameworks:**These three articles focused on the class syllabus and two different ways to implement grading.- Priscilla Bremser, What’s in Your Syllabus?
- Kate Owens, A Beginner’s Guide to Standards Based Grading
- Elise Lockwood, Let Your Students Do Some Grading? Using Peer Assessment to Help Students Understand Key Concepts

**Everything else in the classroom:**These are the remaining articles that addressed things we do, or could do, in the classroom. Drew Lewis’ article on social media included a mention of how this helped him learn about Standards Based Grading, listed above.- Drew Lewis, Social Media as a Teaching Resource
- Elise Lockwood, Don’t Count Them Out — Helping Students Successfully Solve Combinatorial Tasks
- Johanna Hardin and Nicholas J. Horton, Preparing the Next Generation of Students in the Mathematical Sciences to “Think with Data”
- Elise Lockwood, Attending to Precision: A Need for Characterizing and Promoting Careful Mathematical Work
- Art Duval, (Don’t?) Make ’em Laugh

**The affective domain.** I was struck by the different articles that explored aspects of the affective domain. Benjamin Braun (our Editor-in-Chief) wrote two articles directly about this, but Taylor Martin and Ken Smith’s article about classroom culture is also largely about what we can do as teachers to structure our classes to help students develop in this direction. Of course, Martin and Smith’s article also goes nicely with the Class frameworks articles above.

- Benjamin Braun, The Secret Question (Are We Actually Good at Math?)
- Benjamin Braun, Believing in Mathematics
- Taylor Martin and Ken Smith, Creating a Classroom Culture

**Student voices. **Once again, we featured several articles written by students giving their different perspectives. A.K. Whitney wrote about beginning math courses, Sabrina Schmidt about her undergraduate math major overall, and Steve Balady about the program he started as a graduate student.

- A. K. Whitney, Shredding My (Calculus) Confidence
- Sabrina Schmidt, What I Wish I Had Learned More About in College Mathematics
- Steve Balady, We Started a Directed Reading Program (And So Can You!)

**K-12.** Although our main focus is on undergraduate mathematics teaching and learning, it is neither possible nor wise to put a rigid barrier between K-12 and post-secondary. All of these articles find some connection or another between these two levels, whether through curriculum, outreach, or teacher preparation.

- Erin Baldinger, Shawn Broderick, Eileen Murray, Nick Wasserman, and Diana White, Connections between Abstract Algebra and High School Algebra: A Few Connections Worth Exploring
- Matt Baker, Number Theory and Cryptography: A Distance Learning Course for High School Students
- Kathleen Fowler, Start Small, Think Big: Making a Difference Through K-12 Mathematics Outreach
- Jennifer S. McCray, What is Early Math and Why Should We Care?

**Policy, etc.** These are articles that are more broad than a single classroom, and report on, or advocate for, changes that can be made to curriculum and beyond. The latter two articles include actions that ordinary mathematicians and mathematics instructors can take, mostly aimed at the K-12 level.

- Benjamin Braun, Recent Reports and Recommendations Related to Courses in the First Two Years of College Study
- Art Duval, Kristin Umland, James J. Madden, and Dick Stanley, Wanted, Mathematicians for an Important but Difficult Task
- Priscilla Bremser, Imagining Equity

**And one more thing**. Not fitting into any one other category was the article collecting the varied personal reflections about this year’s Joint Math Meetings by each of the members of the editorial board.

I cannot accept that mathematics be taught in a vacuum. Yes, mathematics is beautiful, be it pure or applied. However, in our age of immediacy for students we need to move more of our efforts to teaching mathematics in context, in touch with the real world. We should incorporate more modeling and applications in our mathematics courses to richly support and motivate our students in their attempts to learn mathematics and we should support colleagues who seek to use this approach.

Over the course of time I have moved to this position. At first I used applications of mathematics in course lectures, e.g., error correcting codes in algebra, cryptology in number theory, life sciences in calculus, and engineering in differential equations. Then I assigned students to read articles in other disciplines and share these applications in class. Finally, I incorporated projects in which students could see and practice the application of mathematics. Introducing a modeling scenario makes the mathematics immediate; what do I do right now? Students desire to address the problem at hand, which is real to them, primarily because it intrigues them and piques their curiosity. Thus the mathematics becomes a necessary tool they are ready to learn. I eventually used the application to motivate the learning of the mathematics *before* introducing that mathematics. This is a “flipping” of content.

Some students are a bit shy, even resistant, to this approach. However, in an active and supportive learning environment in which students work in small groups and the teacher works the room by watching, visiting, listening, and assisting the groups, students do amazing things. Sometimes they get off a workable track, but colleagues and teachers bring them along. Students make mistakes, but as we know, learning from mistakes is an important part of learning [BrownEtAl2014]. Indeed, we do it all the time ourselves and call it conjecture and research.

**Practicing What is Preached**

For some time, many colleagues have been calling for using modeling in the mathematics curriculum, be it after the introduction and practice of the mathematical topics or before the mathematics is introduced. An example of the latter is to give elementary school students objects – lots of them – and ask them to describe what they have. Quite often, and quite naturally, they will settle on one attribute, e.g., color, weight, size. The vehicle for description is usually an organized list and quite often an associated visual; something we would recognize as a histogram. We need not formally introduce the notion of a histogram; rather just name it after our students invent it for their immediate purpose.

At a more advanced level, one example [Winkel1997] is to ask students to put one eye on a point on a hillside opposite a mountain across the valley and describe what they can see. In the course of their investigations students invent the notions of partial derivative, tangent plane, and normal to a surface. In another activity [Winkel2008], students invent Fourier series by coming up with the rather natural criteria for measuring best fit of a trigonometric series, motivated by images of spectra from chemistry, voice studies, and seismology. In [LibertiniBliss2016] the authors demonstrate that one can cover traditional topics and techniques in differential equations courses and also introduce rich modeling activities to motivate and consolidate learning. We have found that when students see a modeling situation first, it really motivates the learning of the differential equations material and their grasp of the mathematics is firmer and lasts longer because of the modeling experience. Indeed, Dina Yagodich, of Frederick Community College, says that throughout the semester her students refer to a first day of class activity on death and immigration modeling with simulations using m&m candies [Winkel2014], an indication of the importance and meaning of a modeling first approach to teaching.

**The Big Picture **

In the 2015 CUPM Curriculum Guide to Majors in the Mathematical Sciences, described by Martha Siegel in this AMS Blog [Siegel2015], there is rich support for applications and use of technology in many mathematics courses. From the Course Group on Differential Equations of the CUPM Curriculum Guide material [CUPMODE2015], we note, “There are major applications involving differential equations in all areas of science and engineering, and so many of these should be included in the ODE courses to show students the relevance and importance of this topic.” In the section, “Technology and the Mathematics Curriculum,” of main report [CUPM2015] there is strong encouragement to include technology wherever possible as its use enhances understanding and enables more sophisticated modeling and applications, thereby motivating students.

For years COMAP [COMAP2016] has enriched the repertoire for teachers who seek to motivate mathematics through modeling and application with the production of UMAP Modules, journals, texts, videos, and modeling competitions. I have worked with students who took the Mathematical Contest in Modeling and the Interdisciplinary Contest in Modeling [MCMICM2016] offered by COMAP. This is a four day, team of three, competition in which students apply the mathematics they know and learn lots more mathematics en route to solve a real world problem. Students always say, “This is the best mathematical experience in my life.” Hands down the students tell us that applying mathematics in context and on the spot for the competitions in order to build a model is the most rewarding experience of their undergraduate mathematics.

The Society for Industrial and Applied Mathematics (SIAM) and COMAP have recently released a powerful report, *Guidelines for Assessment and Instruction in Mathematical Modeling Education *[GAIMME2016], in support of modeling throughout the mathematics curricula from K-16. This report is rich in support of why, what, how, and when to both introduce and assess/evaluate modeling efforts in the classroom. The report encourages and supports faculty with little experience in modeling to get into the game and offers practical suggestions and illustrations which should enable more faculty to incorporate modeling in their teaching.

**Taking a Natural Step and Building Community**

We offer an effort to include modeling in one course, differential equations, in the hope that others will join the effort and also do so in other courses. We have created a freely-available community for teachers and students called SIMIODE — Systemic Initiative for Modeling Investigations and Opportunities with Differential Equations. SIMIODE is about teaching differential equations using modeling and technology upfront and throughout the learning process. You can learn more at our dynamic website www.simiode.org [SIMIODE2013] where we offer a community in which colleagues can communicate, contribute, collaborate, publish, teach, explore, etc.

SIMIODE is a teacher repository of materials and references to other useful sources of materials and ideas concerning teaching differential equations using modeling and technology. SIMIODE offers a growing set of Modeling Scenarios. These are key pedagogical components of SIMIODE in which a modeling situation, rich in detail, motivates the study of differential equations. Additionally, there are Technique Narratives which provide techniques and strategies for solving differential equations with motivating examples, activities, and exercises. These materials are double-blind, peer reviewed, and published on-line at SIMIODE. In addition SIMIODE offers videos from which students can collect their own data for modeling with differential equations, both at SIMIODE [SIMIODE2013] and at SIMIODE’s YouTube Channel [SIMIODEYouTube2014].

Examples for learning differential equations with modeling from SIMIODE include such topics as chemical kinetics, sublimation process, Torricelli’s Law, feral cat control, dialysis, word propagation, mixing fish, spread of oil slick, ant tunnel building, pendulum study, machine replacement, pursuit, drug administration, spring mass configurations, shuttlecock fall, design of stadium, hang time, malaria control, electric circuit, whales and krills, and many more. In each case a scenario, often with data, is offered and students are supported in building differential equation models to address the situation. Quite often the model comes before the introduction of the formal differential equation, indeed, the model motivates the mathematics.

SIMIODE is project and inquiry-based learning at its core, for teachers can find (and create and publish their own) activities in which students discover and build differential equation models to address the scenario offered. The use of technology, as appropriate, encourages a rich solution space, addressing technical, graphical, numerical, and symbolic issues in order to demonstrate techniques and address issues for the model under study. Technology permits deeper understanding and richer analyses.

Most importantly, SIMIODE is a community of teachers and students, wherein teachers can collaborate in building modeling opportunities, address issues appropriate to their interests and the interests of their students, and reach out to new colleagues who are interested in teaching differential equations using modeling as the motivation for the subject. Within SIMIODE teachers can build their own course, form groups based on common themes from class rosters to special student teams, and can work on projects with colleagues and students from different campuses.

SIMIODE is sponsoring minicourses at both MathFest in August 2016 and the Joint Mathematics Meetings (JMM2017) in January 2017 as well as conducting a Special AMS Session, “Experiences in teaching differential equations in a modeling first approach,” at JMM 2017. Thus, there is ample opportunity to get first-hand experience in this approach to teaching in addition to collegial support from the on-line community at www.simiode.org.

**Conclusion**

The message is this: Students can learn mathematics in context and we should use mathematical modeling and technology to teach mathematics. The encouragement to use modeling in mathematics coursework and support is growing in the form of resources, professional society encouragement, collegial conversations, and support communities such as SIMIODE.

Just as with radio waves that are everywhere, our receivers tune in and pick up this message about the joy and power of using modeling in mathematics instruction. More importantly, many are engaging due to what they see and hear. We invite you to join us.

**References **

[BrownEtAl2014] Brown, Peter C., Roediger, Henry L., and McDaniel, Mark A. *Make It Stick: The Science of Successful Learning.* Belknap Press, 2014.

[COMAP2016] The Consortium for Mathematics and Its Applications. 2016. www.comap.com. Accessed 17 May 2016.

[CUPM2015] 2015. CUPM. *2015 CUPM Curriculum Guide to Majors in the Mathematical Sciences*. Editor Paul Zorn. http://www.maa.org/sites/default/files/pdf/CUPM/pdf/CUPMguide_print.pdf . Accessed 2 May 2016.

[CUPMODE2015] Devaney, R. 2015. *Ordinary Differential Equations Course Report*. http://www2.kenyon.edu/Depts/Math/schumacherc/public_html/Professional/CUPM/2015Guide/Course%20Groups/OrdDiffeq.pdf . Accessed 5 May 2016.

[GAIMME2016] COMAP and SIAM. 2016. *Guidelines for Assessment and Instruction in Mathematical Modeling Education (GAIMME)*. http://www.comap.com/Free/GAIMME/index.html . Accessed 1 May 2016.

[LibertiniBliss2016] Libertini, J. and K. Bliss. 2016. Using Applications to Motivate the Learning of Differential Equations. To appear in Association for Women in Mathematics publication.

[MCMICM2016] MCM/ICM. 2016. Mathematical Contest in Modeling and Interdisciplinary Contest in Modeling. COMAP. http://www.comap.com/undergraduate/contests/. Accessed 17 May 2016.

[SIMIODE2013] SIMIODE. 2013. Systemic Initiative for Modeling Investigations and Opportunities with Differential Equations. www.simiode.org. Accessed 1 May 2016.

[SIMIODEYoutube2014] 2014. SIMIODE YouTube Channel. https://www.youtube.com/channel/UC14lC-tyBGkDPmUnKMV3f3w. Accessed 1 May 2016.

[Winkel1997] Winkel, B. J. 1997. In Plane View: An Exercise in Visualization*. International Journal of Mathematical Education in Science and Technology*. 28(4): 599-607.

[Winkel2008] Winkel, B. J. 2008. Fourier Series: Optimization Opportunity. *International Journal of Mathematical Education in Science and Technology*. 39(2): 276-284.

[Winkel2014] Winkel, B. J. 2014. 1-1-S-MandMDeathAndImmigration. https://www.simiode.org/resources/132 . Accessed 15 May 2016.

]]>