Quicker way to get large number of submissions

[curtispro]

Hello,

I have a Canvas course with 30 students, but there are 380 assignments.

I am using your canvasapi to get the submissions of each assignment multiple times a day (students can add new submissions throughout the semester)

First I have a list of all assignments

assignments = course.get_assignments()

then multiple times a day I do the following

submissions = {}
for assignment in assignments:
          submissions = assignment.get_submissions()
          for submission in submissions:
                     #collect submission data

The second loop takes a long time time to complete. Setting per_page = 100 in get_submission() isn't improving things, but I don't understand enough of how things work to be able to say why it should. Just a "what have I tried so far" comment.

Is there a quicker way to collect the submission data for all students in all assignments?

[bennettscience]

It takes a long time because each assignment has to check for 30 students, who may have multiple submissions. So, your loop right now is doing the equivalent of 380 * 30, which is 11,400 API calls at minimum. It also depends on what you're doing with the submission data. Is your script processing the submission objet at all? Or are you just writing information to a CSV or database? With each submission, are you using some of those methods to get specific data like comments, media, etc?

Here are a couple things you can try to speed it up a little bit in order from least complex to more.

Process submissions later

If you're doing work with each submission, it might be faster to just download everything into a sheet or SQLite database and then process it for whatever you're looking for. Use this script as a pipeline to get data from Canvas to your machine.

Get all submissions at the course level

You have the option of getting all submissions at the course level without looping through assignments. The Canvas documentation lists params you can use to filter results to active students or submissions following a given date. Be sure to pass the student_ids[]=all variable.

# Set up your Canvas object...
submissions = course.get_multiple_submissions(student_ids='all')  # Returns a paginated list
for submission in submissions:
    # Do stuff here...

Use concurrent.futures for parallel requests

This is a more complex approach, but one I use to run a large report periodically. Last year, we had a course with our 1600 staff members that included 25 assignments. From time to time, I generate a CSV showing a list of missing assignments for staff. So, potentially 3700 requests to get submission data. Using Python's concurrent.futures to run tasks in parallel, I'm able to get a full report in ~5 minutes.

This is a good intro to futures if you want to read more.

Here's a sample script you can use as a model if you're interested:

import csv
import re
from canvasapi import Canvas
import concurrent.futures  # run parallel tasks
from functools import partial  # define a function to run for each user

# Set up a CSV writer
writer = csv.writer(open("your_csv.csv", "w"))

def main():
    canvas = Canvas('URL', 'KEY')  
    course = canvas.get_course(12345)

    enrollments = course.get_enrollments(state='active', type='StudentEnrollment')  # skip teachers and observers
    
    # set up the environment. Run 3 processes at a time. Could be more, but you might get rate-limited.
    with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
        data = []  # list to hold results
        job = partial(process_user, course=course)

    # This bit is where the speed comes in. For each enrollment, it calls `process_user` passing the course
    # object and the enrollment. That subtask requests information from Canvas and returns data that is
    # stored here. 
    # These results are _pending_ the completion of the request. Since it's pending, Python can move between
    # requests, allowing for faster work on long-running tasks.
    results = [executor.submit(job, enrollment) for enrollment in enrollments]
    
    # A `future` in Python is a request that hasn't resolved yet. You can use it as an object to track progress.
    # As each future resolves (returns data), the data is returned to the `data` list we started at the beginning.
    # The print statement is just nice for you to see how far you are in the process in the console.
    for f in concurrent.futures.as_completed(results):
        data.append(f.result())
        print(f"Processed {len(data)} in {len(list(enrollments))} in {course})

    # Once everything is done, batch-write the data to your spreadsheet.
    writer.writerows(data)

def process_user(enrollment, course):
    """Handle getting assignments for a single user

    Args:
        enrollment (canvasapi.enrollment.Enrollment): Canvas <Enrollment> object
        course (canvasapi.course.Course): Canvas <Course> object

    Returns:
        [list]: formatted list for writing to the CSV
    """
    # Get the user's missing assignments.
    missing = get_user_missing(course, enrollment.user["id"])

    # Get the user's email address if available
    regex = re.compile("@")

    # The 'Test Student' account has no login, so do a check here
    if 'login_id' in enrollment.user:
        if regex.search(enrollment.user["login_id"]) is None:
            email = f"{enrollment.user['login_id']}@yourdomain.com"
        else:
            email = enrollment.user["login_id"]
    else:
        email = ""

    # Return data to the main process to append to the spreadsheet.
    return [
        enrollment.user["sortable_name"],
        email,
        course.name,
        enrollment.last_activity_at,
        len(missing),
        ", ".join(missing),
    ]


def get_user_missing(user_id, course):
    """Get the missing assignments for the user in a course

    Args:
        user_id (int): user ID to request assignments for
        course (canvasapi.course.Course): Canvas <Course> object

    Returns:
        [type]: [description]
    """
    # Request one user at a time so you can write each row to a spreadsheet
    submissions = course.get_multiple_submissions(
        student_ids=[user_id],
        include=["assignment", "submission_history"],
        workflow_state="unsubmitted",
    )

    # Get a list of assignment titles to write to a sheet using a list comprehension rather than a for loop.
    missing_list = [
        item.assignment["name"]
        for item in submissions
        if item.workflow_state == "unsubmitted" and item.excused is not True
    ]

    return missing_list