GatorGrader

The only tool you'll need to ensure your student's code and writing is up to speed!

Quickstart Guide

• Starter Repositories An easy way to get started with GatorGrader is to check out our sample starter repositories. The following starter repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTeX and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of red checks. This is only meant to be a “starter” and give an insight into what must be accomplished within the lab.
• Solution Repositories The next step to get involved with GatorGrader is to checkout our sample solution repositories. The following solution repositories provide examples of how GatorGrader files should be created to check programs and documentation for different languages: Java, Python, LaTex , and HTML with CSS. These examples also show how to integrate GatorGrader with GitHub Classroom and Travis CI. When you follow these examples, the Gradle plugin for GatorGrader will install it automatically when you run gradle grade in a terminal window. Please be aware that these repositories are meant to have a majority of green checks. This is meant to show what must be accomplished within a lab/practical and what it looks like when those tasks are completed.

Key Features

GatorGrader automatically checks the work of technical writers and programmers. It can:

• Enable GatorGrade to check projects implemented and documented in a wide variety of languages (e.g., Java, Python, LaTeX, Markdown, HTML, and CSS).

• Integrate with GitHub Classroom to check solution and starter repositories created for professors and students, respectively.

• Run in a cloud-based environment like Travis CI or on the command-line of a developer's workstation.

• Operate as a "batteries included" grading tool, supporting automated checks like the following:

  • Does a file exist in the correct directory with the requested name?

  • Does technical writing contain the desired number of words and paragraphs?

  • Does source code contain the designated number of language-specific comments?

  • Does source code or technical writing contain a required fragment or match a specified regular expression?

  • Does a command execute correctly and produce the expected number of output lines?

  • Does a command execute and produce output containing a fragment or matching a regular expression?

  • Does a GitHub repository contain the anticipated number of commits?

Aligning with key recommendations in a recent National Academies report, GatorGrader helps instructors automatically check student submissions in both introductory and application-oriented classes using languages like Markdown, Java, Python, JavaScript, CSS, and HTML. GatorGrader does not aim to solve problems related to building and linting a project or managing an assignment's submission, instead integrating with existing tools and systems like Gradle, GitHub, and GitHub Classroom to effectively handle those tasks.

Installing GatorGrader

Installing GatorGrader is not necessary if you intend to use it through GatorGrade. If you want to participate in the development of GatorGrader, you should first install Git to access the project's GitHub repository and uv for its support of fast package and virtual environment management. After completing the installation of these tools, you can type the following command in your terminal window to clone GatorGrader's GitHub repository:

git clone https://github.com/GatorEducator/gatorgrader.git

If you plan to develop new features for GatorGrader or if you want to run the tool's test suite in Pytest, then you will need to install a virtual environment for development by typing uv sync in the directory that contains GatorGrader. For help with this process, refer to uv's documentation.

Testing GatorGrader

Automated Testing

The developers use Pytest for the testing of GatorGrader. Depending on your goals, there are several different configurations in which you can run the provided test suite. If you want to run the test suite to see if the test cases are passing, then running this command in a terminal window will perform testing with the version of Python with which uv's virtual environment was initialized.

uv run task test

Test Coverage

Along with running the test suite, the developers of GatorGrader use statement and branch coverage to enhance their testing activities. To see the coverage of the tests while also highlighting the lines that are not currently covered by the tests, you can run this command in a terminal window. As with the previous command, this will run the tests in the version of Python with which uv's virtual environment was initialized.

uv run task cover

Testing with Multiple Python Versions

The previous two commands run the test suite in the version of Python with which uv was initialized. If you want to run the test suite against a different Python version, you can switch uv's Python version before running the task:

uv python pin <python-version>
uv run task test

For example, to test with Python 3.12:

uv python pin 3.12
uv run task test

Code Linting

The developers of GatorGrader use Ruff for linting and code formatting, along with additional tools like bandit for security analysis, radon for complexity metrics, and xenon for code quality thresholds. After installing GatorGrader's development dependencies with uv, you can run all of the linters by typing this command in a terminal window:

uv run task lint

You can also run individual linters as needed:

uv run task lint-ruff       # ruff linting
uv run task lint-ruff-format # ruff formatting check
uv run task lint-bandit     # security analysis
uv run task lint-xenon      # code quality thresholds

Automated Checks

Want to learn about our linting checks? Check us out on our website, GatorGrader! We have detailed descriptions of our linting checks and more! To get an idea of the linting checks we offer, here is a quick list:

1. ConfirmFileExists

2. CountCommandOutput

3. CountCommits

4. CountFileLines

5. CountFileParagraphs

Want to learn about our automated checks? Check them out on our website, gatorgrader.org! We have detailed descriptions of our automated checks and more!

Something you should know when working with our checks is that all of them come with some optional arguments. Optional arguments that you are likely to encounter:

• -h
• --help
• --exact
• --advanced

If --help is tagged along with a check then a help message will be displayed and then exited. If further assistance is needed, please contact us on GitHub.

Another feature with our automated checks is the plug-in based approach. This allows users to implement their own check if our initial checks do not fulfill a check that you find necessary.

Running GatorGrader

Students and instructors normally use GatorGrader through GatorGrade, specifying the requested checks in a gatorgrade.yml file. When run through GatorGrade, GatorGrader reports each check that it performed, additionally sharing a diagnostic message for each check that did not pass. Individuals who want to run GatorGrader as a stand-alone Python application can install GatorGrader itself through Pipx, and then run gatorgrader --help to get more details on the command-line interface.

Instructors often run GatorGrader in conjunction with other tools that check source code and technical writing. For instance, in a Java-based introductory course, instructors could verify student submissions with Checkstyle, thereby ensuring that the Java source code adheres to the requirements in the Google Java Style Guide. In this course, an instructor could require that Markdown files with technical writing meet the standards described in the Markdown Syntax Guide, meaning that all Markdown files must pass the checks performed by the Markdown linting tool. These assignments could also require that all submitted technical writing must adhere to the standards set by the Proselint tool. Since GatorGrader can run an arbitrary command and check its error code, it is also possible to integrate it with a wide variety of other linters, code formatters, and testing tools.

Instructors may at times need to see a full list of checks to have a better understanding and therefore, we feel that it is important to know that there is an easy way for that to happen. This action will be completed through command line and therefore, you can type gatorgrader ListChecks into your terminal, if you've installed GatorGrader as detailed above. This allows for all of the checks to be printed out as output. This output will have the necessary name labeled with the required and optional arguments. If this output does not give enough content, we warmly invite you to navigate to our website, where we go into more detail about our Automated Checks.

Using Docker

Note: Docker supports the older GatorGrader execution tool, GatorGradle. It is not needed for GatorGrade!

A vital part of our process for GatorGrader is to implement and use new techniques to further our tool to grow. This is why we chose to use Docker! Docker is a container platform and therefore, allows students using GatorGrader to just open a container and have easy access to run all commands that would allow them to build, run, and grade their labs and practicals. Docker is an industry standard and therefore, gives us an advantage. To open a container that will allow for the use of GatorGrader, run the following command in your terminal window:

docker run -it --rm --name dockagator \
  -v "$(pwd)":/project \
  -v "$HOME/.dockagator":/root/.local/share \
  gatoreducator/dockagator /bin/bash

From here, you are set! Test it out by building, running, or grading your lab/practical! If you would like to learn more about Docker, please follow this link.

Comparison to Other Tools

Other automated grading tools include:

• autograde-github-classroom: "scripts to download and grade submissions to Github Classroom"
• check50: "a tool for checking student code"
• Classroom Assistant: "desktop application to help you get student repositories for grading"
• nbgrader: "a system for assigning and grading notebooks"
• nerfherder: "scripts for automating grading with GitHub Classroom and Moodle"
• Submitty: "homework submission, automated grading, and TA grading system"
• WebCat: "all-in-one plugin for full processing and feedback generation"

Designed for instructors who want an alternative to simple scripts or stand-alone platforms that do not integrate with industry-standard infrastructure like GitHub and Travis CI, GatorGrader is a tool that automatically checks the work of technical writers and programmers. Unlike other systems, GatorGrader provides a "batteries included" approach to automated grading that makes it easy for instructors to combine well-tested checks for projects that students implement in a wide variety of programming languages. GatorGrader's developers take its engineering seriously, maintaining standards-compliant source code, a test suite with 100% statement and branch coverage, and top-notch source code and user documentation.

Presentations

GatorGrader's creators give presentations about the development, use, and assessment of the tool. Please contact one of the developers if you would like to feature a presentation about GatorGrader at your technical conference. The following list includes some of our team's recent presentations:

• A Hands-on Guide to Teaching Programming with GitHub, Travis CI, and Python
  at PyOhio 2018
• Using GitHub, Travis CI, and Python to Introduce Collaborative Software Development
  at PyCon Education Summit 2018
• Using Python, Travis CI, and GitHub to Effectively Teach Programming
  at PyGotham 2018

Contributing

Are you interested in contributing to GatorGrader, GatorGradle, or any of the sample assignments (e.g., Java, LaTeX, or HTML with CSS)? Great, because we appreciate the involvement of new contributors! Before you raise an issue or start to make a contribution to GatorGrader's repository, we ask that you review the project's code of conduct and the contribution guidelines.