Woops, I zoned out for another year. Thankfully I’m pretty sure no one actually reads most of this stuff, except maybe CS186 students who wanted to look into that one TA who doesn’t teach a live section and who primarily answers debugging questions at 4 AM. Which brings us to today’s topic!
If this is the first thing of mine you’re reading, then hi! My name isn’t important, and I’m nobody in particular. I’ve been teaching stuff in semi-official capacities since I was a high school freshman, when I volunteered at local elementary schools to help run after school math programs and keep kids from burning down bridges, or whatever it is that elementary schoolers would do without daycare. Teaching was also my first real job (real := having to pay taxes) when I worked at Mathnasium, an after school math tutoring/self-studying/getting-ahead/SAT prep/daycare place type thingy. Later I became a lab assistant during my second and third semester for the intro to CS classes at Berkeley, and eventually as a TA up until my graduation a couple of months ago.
Well, okay, most of these aren’t going to be real paradoxes as much as they are catch-22’s or complaints, although I did leave a real paradox in my bio on the class website. For background, from Spring 2020 to Spring 2021 I taught CS 186, Berkeley’s databases course of roughly 600 students per semester. Starting in Fall 20 I was a head TA, which is a fancy way to say I answered more emails and worked extra hours.
An important piece of context is that the course’s assignments are essentially a term-long project implementing various components of a database: query optimizer, indices, concurrency control, etc… Three of my main goals as a TA were:
- Decrease friction on the student’s end. A bit of a conflicting goal to go for, after all falling into debugging session rabbit holes or stumbling waste deep in a flawed design is an important thing to be aware of when writing software. But, the primary goal of the assignments were to help students grok the algorithms presented in lecture, and getting caught in the weeds wasn’t conducive to that goal.
- Help students help themselves. There’s a lot of students compared to TAs, and that ratio isn’t getting any better thanks to budget cuts. On top of that, the pandemic made it difficult for students in certain timezones to get help directly. So a nice way to alleviate both issues was to make it easier for students to work out problems on their own.
- Reduce TA workload, again thanks to budget cuts.
While I think I did a pretty good job at hacking away towards these goals, there were a few counterintuitive observations made along the way.
Paradox 1: Making stuff less confusing lead to more confused students?
Bugs and inconsistencies naturally crop up over time, and the programming assignments for 186 were no exception. In the semesters previous to when I started working almost every term a new assignment would be introduced, which would take time away from fixing problems in existing assignments. Or in the words of Vonnegut, “Everyone wants to build, but no one wants to maintain.” After every project I started taking time to scan through common problems and try to iron things out a bit. This could entail:
- Adjusting the given APIs to be less foot-shooty
- Adding tips in the documentation and specification with advice for common errors
- Leaving in more structure in the existing code
- Introducing very, very detailed error output on unit tests
So, after three semesters worth of these changes the amount of confusion decreased, right? Sort of. Overall, projects went a lot smoother – there were significantly less questions on Piazza, the Q&A platform for the class. Survey results showed a drop in complaints about workload and median time spent on assignments went down. But the amount of confusion in office hours became worse. Huh, weird. This was almost certainly due to sampling bias. Consider the two types of students who might come to office hours for help on a programming assignments:
- A student who understands the material well, but is confused about the assignment because of poor documentation, confusing APIs, or ambiguous specifications
- A student who is confused about the material, and subsequently extra confused about the assignment.
Type 1 students used to make up the majority of office hour tickets, and were the type I thought I could target with improvements to help reduce workload on the TA end. And ultimately, helping those students worked. Even when Type 1 students came in it was usually just a matter of pointing out parts of the documentation or the material to rereview – easy!
Helping Type 2 is a bit more involved, since confusions about the assignments would usually be rooted in deeper misunderstandings about the concepts in the course – rather than pointing towards material to rereview, fixing the problem would require an ad hoc lesson, followed by a review of their code to fix anything that was written on false assumptions, all before even getting to the original bug! By cutting down on Type 1 students, the proportion of Type 2 students increased, creating the illusion of students being more confused.
Paradox 2. Removing 60% of students would have cut wait times by… 0%
One of the “quality of life” metrics for a course is how long someone has to wait to get help. Thanks to the power of software and an international pandemic, it because pretty easy to measure by means of an online ticket queuing system. The online system also made it easier to analyze information about where tickets were coming from.
The above graph shows number of tickets per student in descending order, and is a classic zipf distribution – let
T be the number of tickets created by the most frequent student. Then the 2nd most frequent student generated roughly half of
T, the 3rd most generated roughly one third of
T, etc… Ultimately we found that semester that the top 1% of students accounted for 35% of the tickets, while the bottom 80% accounted for about 7.5%. A least it’s still better than the distribution of wealth!
Paradox 3. The duality of man
As mentioned before, many of the changes I introduced as a TA were aimed at making the projects less bad, which often times included changes that closed off certain pitfalls at the expense of freedom in design choice. These weren’t really changes I was happy with – my favorite assignments at Berkeley were the open-ended ones, and when I took the course I never really ran into issues coming up with implementations. But, anonymous feedback made it pretty clear that a non-trivial portion of the class was getting caught in design ruts, and caused many to give up on assignments prematurely or skip them outright. It could argued that these are acceptable losses, but to me it was preferable to sacrifice some difficulty for higher completion rates – the difficulty was secondary to the main goal of having students apply the material.
While historically feedback on assignments was skewed towards “too difficult! make it easier,” by my third semester teaching we hit about 90% “too hard!” and 10% “too easy” – it would be tricky to please everyone. A few ways to do it would be:
- Have students work out mandatory design docs before programming, and get review from TAs. This is a bit foiled though by our already stretched staff allocation.
- Have open-ended extra credit assignments that do allow for more student creativity. This is in the works, probably, since it’s on the list of cans I’ve kicked to future generations of software TAs.
Even with those changes though, I imagine we’d still be somewhere where complaints come from both ends of the spectrum. Which is a good spot to be in! The class’s Fall 20 professor based their speed of lecturing at the point when compaints about being too fast and too slow hit equilibrium. Not being able to satisfy everyone is hardly a paradox, just a fact of life. The real paradox here is one of a particular piece of feedback received along the lines of “the project is too difficult” and eventually concluding in “don’t make it any easier!” You can’t satisfy everyone, and sometimes you can’t even satisfy one!
Paradox 4. Good news is no news
The last “paradox” is more of just a psychological observation, but one that I think is important to acknowledge, especially for other TAs who might find themselves falling into the same pattern. A large part of what I think makes CS 186 a relatively good (or perhaps just a “not bad”) course is that we rely heavily on student feedback. This is what motivated most of the things I worked on as a TA. A common theme I noticed with feedback was that as teaching staff we mainly focused on the negatives – our job was to fix things that weren’t working, and positive feedback didn’t provide any hints towards that end. Positive feedback could sometimes be perceived not as “you’re doing good” but as “you’re not doing anything wrong.”
I think in retrospect, if I’d spent more time focusing on positive feedback instead of micro-optimizing for the negative, I could have enjoyed my time teaching a lot more and cut down on stress. So, some quick advice: good news is good news! By no means should you always be resting on your laurels, but it’s beneficial to take a break now and then :)