text: In this paper we promote introducing software verification and control flow graph similarity measurement in automated evaluation of students' programs. We present a new grading framework that merges results obtained by combination of these two approaches with results obtained by automated testing, leading to improved quality and precision of automated grading. These two approaches are also useful in providing a comprehensible feedback that can help students to improve the quality of their programs We also present our corresponding tools that are publicly available and open source. The tools are based on LLVM low-level intermediate code representation, so they could be applied to a number of programming languages. Experimental evaluation of the proposed grading framework is performed on a corpus of university students' programs written in programming language C. Results of the experiments show that automatically generated grades are highly correlated with manually determined grades suggesting that the presented tools can find real-world applications in studying and grading.
  @cite_11: A system for automatic grading of programming assignments is described here. This grading system consists of a suite of Perl and Java programs, linked by a database, and driven by an interactive, user-grader controlled grading process. The rationale for the process, the process itself-and the process's interaction-with the programs, the database, and the user himself are discussed. This grading system has the ability to discover and accommodate unanticipated solutions by means of a grading process that is highly interactive with its user-the grader. It also has the ability to automatically accommodate a wide range of simple student errors, which can easily befuddle a more naive grading system. This system has been used since the Spring 2000 semester at Rutgers University, as a core part of the students' performance evaluation activities, in the Introduction to computer science course, where it routinely grades some 300-600 weekly assignments.
  @cite_4: 
  @cite_15: 
  @cite_36: The objective of this study is to show that computer tutorial supplements enchance and shorten the learning process in teaching statistics. An introductory statistics course may be separated into three phases: (a) basic fundamentals, (b) statistical concepts, and (c) applications. Computer tutorials allow more time to be spent on the applications phase of the course. Students, retain more when shown applications. They also tend to favor application courses. Hence, statistics courses become more efficient and popular with tutorial supplements. The results of this study may also have positive implications for online teaching.
  @cite_10: While the usage of unit-testing frameworks such as JUnit has greatly increased over the last several years, it is not immediately apparent to students and instructors how to best use tools like JUnit and how to integrate testing across a computer science curriculum. We have worked over the last four semesters to infuse testing and JUnit across our curriculum, building from having students use JUnit to having them write their own test cases to building larger integration and use case testing systems to studying JUnit as an example of good application of design patterns. We have found that, based on this increased presentation and structuring of the usage of JUnit and testing, students have an increased understanding and appreciation of the overall value of testing in software development.
  @cite_9: 
  @cite_1: A discussion is given of alterations that were made to a typical university operating system to record the results of programming exercises in three different languages, includeing assembly language. In this computer-controlled grading scheme provision is made for testing with programmer-supplied data and for final runs with system-supplied data. Exercises run under the scheme may be mixed with other programs, and no special recognition of exercises by the operators is necessary.
  @cite_32: Automatic assessment of programming exercises has become an important method for grading students' exercises and giving feedback for them in mass courses. We describe a system called Scheme-robo, which has been designed for assessing programming exercises written in the functional programming language Scheme. The system assesses individual procedures instead of complete programs. In addition to checking the correctness of students' solutions the system provides many different tools for analysing other things in the program like its structure and running time, and possible plagiarism. The system has been in production use on our introductory programming course with some 300 students for two years with good results.
  @cite_0: 
  @cite_27: An automatic grading system is presented for grading assignments in scientific computing. A student can interactively use this system to check the correctness of his program assignments. The grade for a correct solution is automatically recorded. This paper also considers the security problems with such an automatic grading system.
  @cite_31: Systems that automatically assess student programming assignments have been designed and used for over forty years. Systems that objectively test and mark student programming work were developed simultaneously with programming assessment in the computer science curriculum. This article reviews a number of influential automatic assessment systems, including descriptions of the earliest systems, and presents some of the most recent developments. The final sections explore a number of directions automated assessment systems may take, presenting current developments alongside a number of important emerging e-learning specifications.
  @cite_13: The Quiver (QUIz VERification) System is an Internet server for building, maintaining, and administering programming quizzes. It is similar to the online judges used for programming contests but differs in that it targets the classroom use of programming quizzes as a teaching aid and evaluation tool. It can provide very detailed feedback regarding quiz behavior so that the student can debug her program. This system is developed as part of the grant "Intra-Curriculum Software Engineering Education" funded by the National Science Foundation (DUE 0127439).
  @cite_25: This paper describes a system for the automated assessment of GUI-based programs. This uses the JEWL library, a set of Java packages designed for use by complete beginners to Java which allows students to build GUI-based programs from the 'Hello world' stage onwards. This paper describes how JEWL has also been designed so that it can be used to assess such programs automatically.
  @cite_17: This article describes a model and implementation of a system for automatically testing, evaluating, grading and providing critical feedback for submitted programming assignments. Complete automation of the evaluation process, with proper attention towards monitoring studentâ€™s progress and performing a finegrained analysis is addressed. The tool provides online support to both the instructors and students and is designed for service-oriented integration with a course management system using Web services.
summary: Automated testing is the most common way of evaluating students' programs @cite_31 . Test cases are usually supplied by a teacher and or randomly generated @cite_17 . A lot of systems use this approach, for example, PSGE @cite_1 , Kassandra @cite_27 , BOSS @cite_9 , WebToTeach @cite_15 , Schemerobe @cite_32 , TRY @cite_36 , HoGG @cite_4 , BAGS @cite_11 , on-line Judge @cite_0 , JEWL @cite_25 , Quiver @cite_13 , and JUnit @cite_10 .