Improve custom resolver docs

docsrc/developers/custom.rst

@@ -0,0 +1,133 @@
+.. custom:
+
+Customizing Pedal
+=================
+
+There are a few major ways to extend Pedal with new functionality.
+
+
+Custom Environments
+-------------------
+
+Custom Tools
+------------
+
+Custom Resolvers
+----------------
+
+Technically, all that you need to do to create a custom resolver is to create a function that takes in a ``Report`` and returns a ``FinalFeedback`` object.
+Then attach the ``make_resolver`` decorator to the function.
+It's also a good idea to call the ``report.finalize_feedbacks()`` method to ensure that all the feedback objects have been created before you start working with them.
+
+.. code-block:: python
+
+    from pedal.resolvers.core import make_resolver
+    from pedal.core.report import Report, MAIN_REPORT
+    from pedal.core.final_feedback import FinalFeedback
+
+    @make_resolver
+    def my_resolver(report: Report = MAIN_REPORT) -> FinalFeedback:
+        report.finalize_feedbacks()
+        # ...
+        return FinalFeedback()
+
+
+Of course, the actual work of populating the ``FinalFeedback`` object is up to you.
+You can use the ``report`` object to access the ``Feedback`` objects that have been created by the tools, and use that information to create your own feedback.
+
+Let's take a look at what the ``simple`` resolver does, with some details simplified for explanatory purposes:
+
+.. code-block:: python
+
+    @make_resolver
+    def resolve(report=MAIN_REPORT):
+        # Prepare feedbacks
+        report.finalize_feedbacks()
+        feedbacks = report.feedback + report.ignored_feedback
+        feedbacks.sort(key=by_priority)
+        # Create the initial final feedback
+        final = set_correct_no_errors(report)
+        # Process each feedback in turn
+        for feedback in feedbacks:
+            final.merge(feedback)
+        # Override empty message
+        final.finalize()
+        # Keep track of the final object and also return it
+        # IDK how important this is, but you should probably do it too
+        report.result = final
+        report.resolves.append(final)
+        return final
+
+The ``by_priority`` function is a simple key function that returns the priority of a feedback object.
+This largely relies on the ``DEFAULT_CATEGORY_PRIORITY`` dictionary that is defined in the ``pedal.core.feedback`` module.
+
+.. code-block:: python
+
+    # pedal/core/feedback.py
+    DEFAULT_CATEGORY_PRIORITY = [
+        "highest",
+        # Static
+        Feedback.CATEGORIES.SYNTAX,
+        Feedback.CATEGORIES.MISTAKES,
+        Feedback.CATEGORIES.INSTRUCTOR,
+        Feedback.CATEGORIES.ALGORITHMIC,
+        # Dynamic
+        Feedback.CATEGORIES.RUNTIME,
+        Feedback.CATEGORIES.STUDENT,
+        Feedback.CATEGORIES.SPECIFICATION,
+        Feedback.CATEGORIES.POSITIVE,
+        Feedback.CATEGORIES.INSTRUCTIONS,
+        Feedback.CATEGORIES.UNKNOWN,
+        "lowest"
+    ]
+
+    # pedal/resolvers/simple.py
+    def by_priority(feedback):
+        """
+        Converts a feedback into a numeric representation for sorting.
+
+        Args:
+            feedback (Feedback): The feedback object to convert
+        Returns:
+            float: A decimal number representing the feedback's relative priority.
+        """
+        category = Feedback.CATEGORIES.UNKNOWN
+        if feedback.category is not None:
+            category = feedback.category.lower()
+        priority = 'medium'
+        if feedback.priority is not None:
+            priority = feedback.priority.lower()
+            priority = Feedback.CATEGORIES.ALIASES.get(priority, priority)
+        if category in DEFAULT_CATEGORY_PRIORITY:
+            value = DEFAULT_CATEGORY_PRIORITY.index(category)
+        else:
+            value = len(DEFAULT_CATEGORY_PRIORITY)
+        if priority in DEFAULT_CATEGORY_PRIORITY:
+            value = DEFAULT_CATEGORY_PRIORITY.index(priority)
+            priority = 'medium'
+        offset = priority_offset(priority)
+        return value + offset
+
+
+The ``set_correct_no_errors`` function is from the ``pedal.core.final_feedback`` module,
+and is a convenience function that creates a new ``FinalFeedback`` object with the
+``correct`` field set to ``True``, a blank title and message, and feedback that basically says "No feedback was provided".
+The expectation is that this information will be completely overwritten by other feedback.
+
+Most of the actual logic for "merging" the individual feedback objects and then finalizing the result is
+in the ``FinalFeedback`` class itself.
+This class is *very* similar to the ``Feedback`` class, but it has fewer features, is meant to be serializable
+to a JSON object, and has a few extra methods for merging and finalizing the feedback.
+
+Your custom resolver might not take advantage of the ``merge`` method, but it's still very useful to review
+what that function does in order to understand how Pedal currently defaults to handling feedback objects.
+We strongly recommend checking its source code in ``pedal/core/final_feedback.py``.
+
+Note that the ``finalize`` method does rely on ``combine_scores``,
+which is actually fairly complex. Mostly, you should just be aware that it exists and that it is used to combine the scores of the feedback objects.
+
+Custom Feedback
+---------------
+
+Custom Hooks
+------------

pedal/core/final_feedback.py

@@ -69,7 +69,21 @@ def __init__(self, correct=None, score=None, category=None, label=None, title=No
         self.used = []
 
     def merge(self, feedback):
+        """
+        Process a single piece of feedback, and modify ourselves based on the
+        data found there. This includes updating the correctness, the message,
+        the title, and any other attributes.
+
+        Also keep track of the feedback that was considered, any positive
+        feedback, instructional feedback, and system messages.
+
+        If the feedback was ultimately not meant to be used, we return None.
+        Otherwise, we return the feedback object to indicate that we have
+        incorporated it into our final feedback.
+        """
         self.considered.append(feedback)
+        # Check if we should suppress this feedback based on its
+        # category and label (and also potentially fields)
         category = feedback.category.lower()
         if category in self.suppressions:
             if True in self.suppressions[category]:
@@ -87,6 +101,7 @@ def merge(self, feedback):
                         else:
                             # Oh hey, a match, let's skip this guy
                             return
+        # Check if this label has been suppressed specifically
         if feedback.label in self.suppressed_labels:
             list_of_fields = self.suppressed_labels[feedback.label]
             # Go through each of the sets of fields
@@ -97,41 +112,60 @@ def merge(self, feedback):
                         break
                 else:
                     return
+        # Parse the feedback for its settings
         correct, partial, message, title, data = parse_feedback(feedback)
+        # If it's a system message, just add it to the systems feedback list
         if feedback and feedback.category == Feedback.CATEGORIES.SYSTEM:
             self.systems.append(feedback)
+        # Resolve the score, if present, making sure to handle valence
         if not feedback.unscored and feedback.score is not None:
             # Triggered Negative leads to opposite behavior for operator
             # Also untriggered positive feedback
             invert_logic = ((feedback.valence != feedback.NEGATIVE_VALENCE) == (not feedback))
             inversion = "!" if invert_logic else ""
             self._scores.append(f"{inversion}{partial}")
             feedback.resolved_score = Score.parse(f"{inversion}{partial}").to_percent_string()
+        # If this is was not triggered but had an else message, then add it to the positives list
         if not feedback and feedback.else_message:
             self.positives.append(feedback)
             return feedback
+        # If this feedback was not activated, or was muted, then don't do anything with it
         if not feedback or feedback.muted:
             return
+        # If this is explicitly a positive feedback, add it to the positives list and stop
         if feedback.kind == Feedback.KINDS.COMPLIMENT:
             self.positives.append(feedback)
             return feedback
+        # If this is explicitly instructional, add it to the instructions list
         if feedback.kind == Feedback.KINDS.INSTRUCTIONAL:
             self.instructions.append(feedback)
+        # If this feedback was correct, then update the correctness
+        # Once we have a single incorrect feedback, the whole thing is incorrect
         self.success = self.correct = correct and self.correct
+        # If we don't have a message yet, then use this one
         if message is not None and self.message is None:
             self.message = message
             self.title = title
             self.category = feedback.category
             self.label = feedback.label
             self.data = data
             self.used.append(feedback)
+        # All done, return the feedback out of politeness
         return feedback
 
     def finalize(self):
+        """
+        Finalize the feedback object, setting the title and message to
+        defaults if they are not already set. Also, combine the scores
+        into a single score.
+        """
+        # If we don't have a message yet, then use the default message
         if self.message is None:
             self.title = self.DEFAULT_NO_FEEDBACK_TITLE
             self.message = self.DEFAULT_NO_FEEDBACK_MESSAGE
+        # If we have suppressed correctness, then update that flag
         self.hide_correctness = self.suppressions.get('correct', self.suppressions.get('success', False))
+        # As long as we are allowed, change the default message to the "correct" message
         if (not self.hide_correctness and
                 self.label == self.DEFAULT_NO_FEEDBACK_LABEL and
                 self.category == Feedback.CATEGORIES.COMPLETE):
@@ -141,7 +175,9 @@ def finalize(self):
             self.score = 1
             self.success = self.correct = True
         else:
+            # If they weren't correct, we need to combine the scores
             self.score = combine_scores(self._scores)
+        # Update the success/correct flags
         self.success = self.correct = bool(self.correct)
         return self
 

pedal/resolvers/readme.md

@@ -1,3 +1,4 @@
 # Resolvers
 
-A tool for selecting and managing reported data from other tools, in order to select a relevant piece of feedback.
+A tool for selecting and managing reported data from other tools, in order to select a 
+relevant piece of feedback.