Writing Good Questions for the Internet Era

Jeff Suzuki

CUNY Brooklyn

The forced conversion to distance learning in Spring 2020 caught most of us off-guard. One of the biggest problems we face is the existence of free or freemium online calculators that show all steps required to produce a textbook perfect solution.  A student can simply type in “Solve ” or “Find the derivative of ” or “Evaluate ” or “Solve ,” and the site will produce a step-by-step solution indistinguishable from one we’d show in class.  With Fall 2020 rapidly approaching, and no sign that distance learning will be abandoned, we must confront a painful reality:   Every question that can be answered by following a sequence of steps is now meaningless as a way to measure student learning.

So how can we evaluate student learning?  Those of us fortunate enough to teach courses with small enrollments have a multitude of options:  oral exams; semester-long projects; student interviews.  But for the rest of us, our best option is to ask “internet resistant” questions.    Here are three strategies for writing such questions:

●       Require inefficiency.

●       Limit the information.

●       Move the lines

Require Inefficiency

One of the goals of mathematics education is developing adaptive expertise: the ability to identify which of the many possible algorithms is the best to use on a particular problem.

For example, consider a quadratic equation.  We have at least two ways of solving quadratic equations:  by factoring; or by the quadratic formula.  Which do we use?  Since the quadratic formula always works, there’s no obvious reason why we would ever want to use anything else.  But sometimes using the quadratic formula is like using a chainsaw to cut a dinner roll:  we wouldn’t use it on “Solve $(3x-7)(2x+5) = 0$ ,” and we probably wouldn’t use it on “Solve $x^2-9 = 0$,” though we’d almost certainly use it on “Solve $6x^2 – 19x – 36 = 0$.”   The boundary between the problems we’d attempt to solve by factoring and the problems we’d solve using the quadratic formula can’t be taught:  every student has to find it for themselves through firsthand experience.

It should be clear that requiring inefficiency is a possibility every time there is more than one way to solve a problem.  This approach works even better when one method is clearly (to us) less efficient.   Indeed, the least efficient method is one that doesn’t work, and in some ways, requiring inefficiency in such cases may give us more insight into student learning than their ability to solve a problem.

For example, consider the problem:

If possible, solve by factoring:   $x^2 – 3x – 12 = 0$.  If not possible, show why; then solve using the quadratic formula.

Since the quadratic expression is irreducible over the integers, no online calculator will produce a factorization.  Thus, a student can’t simply look up the answer.  More importantly, in order to provide an answer, they must check every possible pair of factors (and show that none of them work).

There’s an added bonus.  On the same exam, we might ask students to factor various quadratic expressions. We argue that a student’s attempt to factor $x^2-3x-12$  will actually reveal more about whether a student understands factoring than the successful factorization of an expression like $6x^2 + 19x – 36$.  Thus, we can omit straight factorization questions (which, in any case, can be “solved” by an online calculator).

Limit the Information

Another way to thwart the use of internet calculators is to provide incomplete data.  For example, Wolfram Alpha can find the derivative of any function—provided you give it the function.  Thus we might ask students to solve problems without giving them equations.

This might sound hard to do, but it’s actually pretty easy.  Since the 1990s, state and national mathematics standards have called for increased use of graphical and tabular representations, so source material is plentiful.  Even the most traditional texts include problems based on interpreting graphical and tabular data.   For example:

Suppose you know $f(3) = 5$  and $f'(3) = -4$ .  Let $h(x) = ln f(x)$ .  Find  $h'(3)$.

While this is an algorithmic question that can be easily answered by invoking the chain rule, doing so relies on correctly interpreting the written statements about the function and derivative values.  As such, it is currently beyond the capability of online calculators.

We can also present data graphically:

The graphs of y = f(x) (solid) and y = g(x) (dashed) are shown:

Find the sign of $(fg)’)(0)$.

Again, this is an algorithmic question that can be answered by invoking the product rule.  However, it relies on being able to extract information from a graph, then make a quantitative argument based on the signs of the functions and their derivatives.

Moving the Lines

Requiring inefficiency and limiting information should be viewed as stopgap measures at best.   Thus, when calculators were first introduced, math teachers insisted on “exact answers,” since the student who returned the answer “1.4142135” instead of $\sqrt{2}$ was clearly using a calculator.  But now, even a \$10 calculator can return “exact answers” like $\frac {3+\sqrt {5}} 2$ , so this distinction is no longer useful as a way of distinguishing between students who used a calculator and students who didn’t.  Similarly, while I’m not aware of any app that allows for the user to select a solution method, or that can read graphical or tabular data, there’s no a priori reason why there couldn’t be one.  This means we need a more powerful method of creating internet resistant questions that can adapt to advances in technology.  This leads to a strategy I call “moving the lines.”

To begin with, it’s important to understand that the problem “Solve  $x^2 – 3x – 12 = 0$” does not exist outside of a mathematics classroom.  So we should ask two questions:

●       Where did this problem come from?  This moves the “starting line,” where the problem begins.

●       Why do we want the solution?   This moves the “finish line,” where the problem ends.

Our long-term goal as mathematics educators should be to shift the lines and turn a sprint into a marathon.

Let’s consider this problem.  What leads to  “Solve $x^2 – 3x – 12 = 0$?”  For that, we might consider some of the basic steps in solving any quadratic equation.    One of those steps is to get the equation into standard form. So our problem “Solve  $x^2-3x-12 = 0$” might have come from “Solve $x^2 – 3x = 12$.”  In fact, you’ve probably asked this question before, specifically to identify students who failed to understand the necessity of getting the equation into standard form.

Now where might we have gotten a problem like that?  We might have gotten it from “Solve $x(x-3) = 12$.”   In fact, you’ve probably asked this type of question as well, to identify the students who failed to understand the zero product property.

Note that we still have an equation that can be dropped into an online calculator, so the next step is important:  What type of question leads to a product equal to a number?  There are many times we multiply two numbers to get a quantity of interest; for example, the product of a rectangle’s length and width gives us the area.  This takes us to the problem:

A rectangle has an area of twelve square feet, and its width is three feet less than its length.  Find the length of the rectangle.

In order to answer this question, a student would have to translate the given information into a mathematical form.   This is beyond the capability of online calculators (especially if, as in this case, the numbers are also spelled out).  If you enter the question into Google, you’ll get examples of similar problems, but no solution, effectively reducing you to your class notes and textbook.  If you’re clever enough to switch numbers for the words, you’ll get an answer—which is incorrect (4 feet).

We can further improve the problem by changing the finish line.  Remember that once a student translates this problem into the equation $x(x-3) = 12$, an online calculator can produce the algebraic solution, showing all the steps.  One way to further blunt the ability of the online calculator to answer all questions is to require another step beyond the mathematical solution.  Thus we should ask why we’d want the answer.

Let’s consider:  we obtain the length (and width, since we know it’s three feet less than the length).  So why would we want the length and width of a rectangle?  There are three obvious possibilities:  to find the rectangle’s area; to find the rectangle’s perimeter; and to find the rectangle’s diagonal.  Since we already know the area, we might want either the perimeter or the diagonal.  So we could ask:

A rectangle has an area of twelve square feet, and its width is three feet less than its length.  Find the perimeter of the rectangle.

Even better:

A homeowner wants to fence a garden in the shape of a rectangle.  The garden must have an area of twelve square feet, where the width is three feet less than its length.  The fence will cost two dollars per foot.  How much will it cost to enclose the garden?

The best part about this approach is that as technology advances, we can shift the lines in response.  Perhaps some day we’ll be able to enter the above problem into a search engine and get the correct answer.  So the next step will be to shift the lines again:  move the starting point further back by imagining where the problem might come from; and move the finish line further forward by considering why we’d want to know the cost.

The Road Ahead

Notice that we end with something that might be called a “real world” problem.  But a homeowner rarely has to build a garden with a specific area and relationship between the sides:  it would be a stretch to call the problem above a real world example of how to use mathematics.

What’s more important is that real world problems don’t come with instructions on how to solve them, so they must be solved inefficiently, by trying different approaches until we find one that works.  Real world problems don’t come with formulas attached to them, so they must be solved without complete information.  And real world problems often change, so we must expect that the starting and finishing lines will change on us.

What this means is that regardless of when or if we can resume traditional resource-restricted exams, we should consider requiring inefficiency, limiting information, and shifting the lines on all our assessments.  Sooner or later, our students will leave our classroom.  If what they learned can be replaced by someone using a free internet app, then they can be replaced by a free internet app.  So it’s not just about making our questions internet resistant:  it’s also about making our students internet resistant.

Addendum

We’re stronger together.  Readers interested in sharing their “internet resistant” questions should email them to me at jsuzuki@brooklyn.cuny.edu, and I’ll put up a selection of these in a later post.