{"id":3689,"date":"2015-03-31T07:01:50","date_gmt":"2015-03-31T07:01:50","guid":{"rendered":"http:\/\/revoscience.com\/en\/?p=3689"},"modified":"2015-03-31T07:01:50","modified_gmt":"2015-03-31T07:01:50","slug":"reviewing-online-homework-at-scale","status":"publish","type":"post","link":"https:\/\/www.revoscience.com\/en\/reviewing-online-homework-at-scale\/","title":{"rendered":"Reviewing online homework at scale"},"content":{"rendered":"

System clusters similar student programs together, so instructors can identify broad trends.<\/strong><\/em><\/span><\/p>\n

\"MIT<\/a>
MIT graduate students Elena Glassman and Jeremy Scott
Photo: Jose-Luis Olivares\/MIT<\/figcaption><\/figure>\n

CAMBRIDGE, Mass. —\u00a0In computer-science classes, homework assignments consist of writing programs. It\u2019s easy to create automated tests that determine whether a given program yields the right outputs to a series of inputs. But those tests say nothing about whether the program code is clear or confusing, whether it includes unnecessary computation, and whether it meets the terms of the assignment.<\/span><\/p>\n

Professors and teaching assistants review students\u2019 code to try to flag obvious mistakes, but even in undergraduate lecture courses, they usually don\u2019t have time for exhaustive analysis. And that problem is much worse in online courses, with thousands of students, each of whom might have approached a problem in a slightly different way.<\/span><\/p>\n

In April, at the Association for Computing Machinery\u2019s Conference on Human Factors in Computing Systems, MIT researchers will present a new system that automatically compares students\u2019 solutions to programming assignments, lumping together those that use the same techniques.<\/span><\/p>\n

For each approach, the system \u2014 called OverCode \u2014 creates a program template, using variable names that a preponderance of students happen to have converged on. It then displays templates side-by-side, graying out the code they share, so the differences stand out in relief. And from any template, instructors can, if they choose, pull up a list of the student programs that accord with it.<\/span><\/p>\n

Instructors who notice variations across templates that make no difference in practice can also write rules establishing the equivalence of alternatives. In some instances, for example, \u201cy*x\u201d might yield a different result than \u201cx*y\u201d, but \u2014 depending on the ways in which x and y are defined \u2014 in other instances, it won\u2019t. When it doesn\u2019t, an instructor could further winnow down the number of templates by creating the rule \u201cy*x = x*y\u201d.<\/span><\/p>\n

The system could allow instructors of online courses to provide generalized feedback that addresses a broader swath of their students. But it could also provide information on how computer-science courses \u2014 both online and on campus \u2014 could be better designed.<\/span><\/p>\n

With online courses, \u201cin a few months, you can have many orders of magnitude of students go through the same material and find all the interesting alternative solutions or make the same errors,\u201d says Elena Glassman, an MIT graduate student in computer science and engineering and first author on the new paper. \u201cThen it\u2019s taking all those records of what people did and making sense of it so that when we run the course again, it\u2019s better, and when we run the course residentially, we\u2019re better able to handle the particular 200 students that we\u2019re meeting with on a regular basis.\u201d<\/span><\/p>\n

Two programs that perform the same computation may have code that looks somewhat different. The programmers may have chosen different variable names \u2014 \u201ctotal,\u201d say, in one case, versus \u201cresult\u201d in the other. Subfunctions may be executed in different orders.<\/span><\/p>\n

So in addition to comparing programs\u2019 code, OverCode observes the values that variables take on as the programs execute. Two programs with variables that take on the same values in the same order are judged to be identical.<\/span><\/p>\n

In their new paper, Glassman and her collaborators \u2014 her thesis advisor, professor of computer science and engineering Rob Miller; her fellow graduate student Jeremy Scott;\u00a0Rishabh Singh<\/span><\/a>, who completed his PhD at MIT last year and is now at Microsoft Research; and Philip Guo, an assistant professor of computer science at the University of Rochester \u2014 also report the results of two usability studies that evaluated OverCode.<\/span><\/p>\n

In the studies, 24 experienced programmers reviewed thousands of students\u2019 solutions to three introductory programming assignments, using both OverCode and a standard tool that displays solutions one at a time. For each assignment, the subjects were given 15 minutes to assess the strategies students most commonly used to design a particular function and to provide general feedback on each, complete with example code.<\/span><\/p>\n

Remarkably, when assessing the simplest of the three assignments, the subjects analyzing raw code performed as well those using OverCode: In both cases, the five strategies they identified covered about half of the student responses.<\/span><\/p>\n

For the most difficult of the three assignments, however, the OverCode users covered about 45 percent of student responses, while the subjects analyzing raw data covered only about 9 percent. \u201cThe strategy starts to shine on more-complicated programs,\u201d Glassman says.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

System clusters similar student programs together, so instructors can identify broad trends. CAMBRIDGE, Mass. —\u00a0In computer-science classes, homework assignments consist of writing programs. It\u2019s easy to create automated tests that determine whether a given program yields the right outputs to a series of inputs. But those tests say nothing about whether the program code is […]<\/p>\n","protected":false},"author":6,"featured_media":3690,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[43],"tags":[],"class_list":["post-3689","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-computer-science"],"featured_image_urls":{"full":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"thumbnail":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0-150x150.jpg",150,150,true],"medium":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0-300x200.jpg",300,200,true],"medium_large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"1536x1536":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"2048x2048":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"ultp_layout_landscape_large":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"ultp_layout_landscape":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"ultp_layout_portrait":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",600,400,false],"ultp_layout_square":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",600,400,false],"newspaper-x-single-post":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"newspaper-x-recent-post-big":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",540,360,false],"newspaper-x-recent-post-list-image":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",95,63,false],"web-stories-poster-portrait":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",639,426,false],"web-stories-publisher-logo":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",96,64,false],"web-stories-thumbnail":["https:\/\/www.revoscience.com\/en\/wp-content\/uploads\/2015\/03\/MIT-Mooc-Grading-01_0.jpg",150,100,false]},"author_info":{"info":["Amrita Tuladhar"]},"category_info":"Computer Science<\/a>","tag_info":"Computer Science","comment_count":"0","_links":{"self":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts\/3689","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/comments?post=3689"}],"version-history":[{"count":0,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/posts\/3689\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/media\/3690"}],"wp:attachment":[{"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/media?parent=3689"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/categories?post=3689"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.revoscience.com\/en\/wp-json\/wp\/v2\/tags?post=3689"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}