Ready Set Research -> 2023 - 2024
Replies: 29 comments
-
|
To begin, I'll recap an ongoing conversation with my first reader that took place over the course of the past week and a half. Generating an SE Comp IdeaMonday, Sept. 4My first meeting with Professor Kapfhammer--who will simply be referred to as GK going forward for the sake of keystrokes--primarily saw me sharing my initial senior comp idea: a browser-based distance teaching platform for imparting programming fundamentals (specifically through the Rust programming language). The intended audience for the platform was adult distance learners, and the main motivation behind the idea was to explore both the impact of gamification (which as a research body is riddled with gaps) and the utility of Rust as an introductory language (despite its reputation for having a steep onboard) while producing a hopefully useful learning platform. Thinking about what separates a Software Engineering comp from a Computer Science comp was the focus of the conversation, and we ultimately determined that the idea--while it certainly had merit as a Computer Science comp--wasn't particularly focused on the desired outcomes for a Software Engineering comp. More specifically, GK stated--and I agreed--that a project emblematic of Software Engineering would have broad utility and modularity (i.e., it would be built for re-use). We agreed that an appropriate project would focus more on the software artifact itself (and the engineering process that created) and less on experimental evaluation, which would've been at the forefront of the original proposed idea. In short, it was back to the drawing board for me. Thursday, September 7After spending some time mulling over my predicament--I didn't have a suitable comp idea!--I met with GK to pitch a new idea: an evaluation tool that would be situated at the top of a monorepo containing all assignments (via Git submodules) for a given course and run customized test suites to not only provide grades, but other metrics that might interest an instructor. Somehow, on paper this idea felt like it had novelty, but the more I discussed it with GK and developed the idea, the greater and more obvious the overlap between my proposed tool and existent tools like gatorgrade became. We affirmed that another important part of a Software Engineering comp was novelty (or as close to novelty as we could get). In order for a software tool to have meaningful utility, it had to do something that wasn't being already being done (or at least not already being done in the Allegheny CIS department). I was clearly interested in creating an artifact that had some educational utility, and GK suggested that there had to be some unique and novel tooling need that might support a fully developed learning platform (which is more or less what I had presented, albeit in two distinct ways). With his suggestion in mind, I went back to the drawing board. Again. Tuesday, September 14In my third meeting with GK I presented my third comp idea, one that seemed like it would have a novel and broadly applicable utility. Following my prior meeting, I had been thinking about abstract syntax tree parsing (a topic in one of the courses I'm currently TLing) as well as what makes certain code complex (or not complex)--a topic that had been in the back of my mind throughout the duration of a summer internship that saw me puzzle through highly complex and difficult-to-maintain code that hadn't been touched in some years. I had done some reading on Halstead metrics, McCabe's cyclomatic complexity, and similar efforts to measure code complexity--such as SonarQube's relatively recent cognitive complexity metric. It was this last metric that had piqued my interest, and the way that the measurement defined increasingly nested code as being increasingly complex. Even then, the complexity SonarQube was measuring was--for lack of a better word--"macro"; it only measured distinct branches of logic across many lines of code. I've never quite been able to wrap my head around lambda expressions in Python. Or list comprehensions for that matter. These often single-line pieces of code are highly complex and--to me anyway (and I assume many introductory programmers)--horribly opaque. But these expressions would have the same level of complexity as a print statement according to SonarQube, Halstead, or whatever other established metric you consulted. Thinking about AST parsing, I wondered if SonarQube's tactic of counting levels of nestedness could be applied on a more "micro" scale--that is, if we could measure complexity by counting the individual "pieces" of logic that were nested within other pieces of logic, so to speak. If this seems horribly abstract, I'll provide a more concrete example of this in a future post (this one's getting lengthy). Here's what I propose: we utilize a tool like tree-sitter that can parse ASTs and use it to measure "micro-complexity". This measurement could then be leveraged in an IDE extension to provide syntax highlighting for over-complex code, or possibly to provide some other data-driven utility (such as an instructor plotting the complexity of their assignments over a semester to ensure an even uptick in complexity). Thankfully, GK agreed that this idea perfectly represents a Software Engineering comp idea. So now, we start building. (And reading and researching.) |
Beta Was this translation helpful? Give feedback.
-
|
This first "research dump" contains a list of relevant articles and research resources pertaining to the topic of Existing Code Complexity Metrics. Research Dump No. 1: Existing Code Complexity Metrics
|
Beta Was this translation helpful? Give feedback.
-
Thursday, September 14My latest meeting with GK saw us spend some time to clearly define expectations related to the SE comp. Those expectations are outlined below, where they will be updated as they continue to be defined over the course of the comp development period. SE RequirementsI'll be using this section to track agreed-upon criteria for an SE comp that might not otherwise be explicitly stated in the CMPSC comp syllabus.
|
Beta Was this translation helpful? Give feedback.
-
|
Unlike my above "research dump", which is used to collect research articles and academic sources, this "technical reference dump" is more interested in technical writing (like tech blog posts, library documentation, etc.) that I suspect will aid me in putting together the software artifact for my comp. Technical Reference Dump No. 1: Using
|
Beta Was this translation helpful? Give feedback.
-
Tuesday, September 19My fifth meeting with GK saw me sharing my progress in implementing the comp, with the most notable piece to share being a feasibility demo of a Rust implementation of Our discussion was brief, but we agreed that the next step was implementing file-reading so that the input could for the parser could be more than a hard-coded snippet of code. Notable steps that I'll want to start investigating (and eventually implementing) will include syntax highlighting and automatic language detection. |
Beta Was this translation helpful? Give feedback.
-
Research Dump No. 2: In Defense of
|
Beta Was this translation helpful? Give feedback.
-
Technical Reference Dump No. 2: Language DetectionOne of the primary reasons that I'm leveraging These reference materials highlight documentation I've dug up related to anything pertaining to programming language detection:
|
Beta Was this translation helpful? Give feedback.
-
Technical Reference Dump No. 3: GitHub ActionsI'm not particularly savvy when it comes to CI/CD--a key part of the software engineering process, and one that I'll need to tackle while building the SE comp. This technical reference dump simply contains documentation that I may use to cobble together my GitHub Actions flow.
|
Beta Was this translation helpful? Give feedback.
-
Tuesday, September 26Another meeting with GK; this particular session I highlighted some of my findings regarding programming language detection, and suggested that this particular feature might be beyond the scope of at least the prototype--a sentiment that GK agreed with. Instead, we decided it'd be appropriate to outfit the program with the capability to parse an additional CLI argument, requiring the user to manually select a language to parse an AST for (rather than automatically detecting this based on the input file selected). A little clunky, but it gets the job done and keeps me from being over-focused on a nice perk rather than the main focus of the comp. We also discussed some additional content to consider adding to my GitHub Actions workflow (which is currently quite bare), and had a brief discussion on selecting an appropriate license for the comp repo. The take-aways from the former discussion have been added into an earlier discussion post about requirements for the SE comp. |
Beta Was this translation helpful? Give feedback.
-
Friday, September 29This is more of an aside, and not directly related to the development of the Senior Comp, but this date marked the Alumin Panel and social at Alden. The panel itself was interesting (if Luis isn't involved in a radio show or podcast then his voice is being wasted) and I spent a large chunk of the social talking with alum Noor about his experience the past year at NetApp. We swapped stories about the respective development cultures of the companies we had worked with (myself drawing from my internship experience at Auto-Owners Insurance). |
Beta Was this translation helpful? Give feedback.
-
Tuesday, October 3After a lengthy meeting with GK primarily focused on brainstorming how to experimentally evaluate my proposed complexity metric, I was left with several big-picture questions to mull over. Namely, should I lean on IRB-approved human research, or develop an additional software tool that can be utilized to do some evaluation via repository mining? Or (if I'm feeling particularly ambitious) some combination of the two? I'm still not sure which way to lean, but there is a pressing deadline coming up (Nov 1, the due date for IRB proposals) that I'll have to keep in mind as I consider which route is best for this project. |
Beta Was this translation helpful? Give feedback.
-
Thursday, October 12Today's meeting with GK had us defining some of the next steps for the comp. We concretely laid out the "story" or selling points to my comp idea in preparation for upcoming pitch presentations, and discussed engineering to-dos that will bring my artifact to a suitable place for said pitch presentation. In broad strokes, these to-dos include:
|
Beta Was this translation helpful? Give feedback.
-
Thursday, October 26Today's meeting with GK had us briefly reviewing content for my pitch presentation tomorrow, and start in on brainstorming the next action item to tackle: measuring cyclomatic complexity. I proposed using the existing AST traversal to grab nodes that might have conditional logic embedded in them, and cross-referencing existing documentation on measuring cyclomatic complexity to determine what types of nodes increment complexity. We agree that implementing this measurement across more than one programming language will likely be a bit of a bear to implement, and after discussing a plan of action today, we devoted our next meeting (on 10/31) to spending the time working together to start to identify cyclomatic complexity incrementing nodes in one language (e.g., Python), and starting the process of comparing measurements (and node identifiers) across other language grammars. |
Beta Was this translation helpful? Give feedback.
-
Tuesday, October 31In my meeting with GK today, we spent some time talking about what the written comp chapters for an SE comp should consider and tackle (as compared to what's laid out for the CS comp chapters); specifically, I'll want to keep in mind actual software artifacts (ideally both open source and enterprise-level) that I can draw on for my related work section. We also discussed some long-term ambitions for the (We admittedly got a little sidetracked during this meeting and spent a good chunk of time talking about a side project--an evaluation tool related to measuring data pertaining to learning objectives in the CIS department.) |
Beta Was this translation helpful? Give feedback.
-
Tuesday, November 7I realized after the fact that I forgot to talk about my last session with GK, so here's a (belated) entry summarizing our time together. As has recently been the trend, a fair chunk of our time was spent excitedly (and distractedly) hashing out thoughts and ideas for the department program outcomes evaluation tool. (An important conversation, but one that's not remotely related to the comp itself.) For the time that we did spend discussing the comp, I demonstrated |
Beta Was this translation helpful? Give feedback.
-
Tuesday, November 14Another slightly belated recap. My time with GK was spent demonstrating some of the SE tooling I added to Additionally, we spent time aspirationally thinking about my work on the comp as well as my contributions to GK's 203 course. He directed me to the PyCon 2024 website and asked me to spend some time thinking about whether I'd be interested in helping with poster efforts for |
Beta Was this translation helpful? Give feedback.
-
Thursday, November 30Since my last meeting with GK, I had started in on the actual writing of the comp. While putting pen to paper, I realized that I had (at best) a fuzzy and not particularly compelling idea of the ethical implications of my project--somewhat problematic since it's meant to play a relatively large role in the chapter content. My time with GK this week was spent bouncing ideas off of each other to bridge this gap, and I walked away with several ideas for ways to further flesh out the ethical implications of complexity metrics. |
Beta Was this translation helpful? Give feedback.
-
Ethics Discussion Dumping GroundThis is simply a place for me to keep my scattered thoughts on productive discussion of the ethical implications of my project:
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @jnormile, thanks for your efforts in documenting your research through this notebook. Overall, the entries that you prepared are very good. Thanks again! I suggest that you could improve some of your entries if you made the |
Beta Was this translation helpful? Give feedback.
-
Wednesday, January 24Today kicked off my meetings with GK for the spring semester. Today we went over the feedback from my chapters GK had provided over the break. Much of the feedback had already been addressed, and after some discussion GK pivoted based on my input; the outstanding to-dos are primarily:
|
Beta Was this translation helpful? Give feedback.
-
Monday, January 29My meeting with GK focused on to-do's pertaining to the actual software tool itself. The main take-aways/priorities for me to focus on consisted of:
|
Beta Was this translation helpful? Give feedback.
-
Friday, February 9Today GK and I briefly discussed the content I had prepared for my draft of Chapter 3, and also talked about some of my thought processes regarding upcoming data collection. Outstanding TODO's include:
|
Beta Was this translation helpful? Give feedback.
-
Monday, February 12Not much new to report since my last meeting had gotten bumped up. This was a brief session with GK, with the conversation focusing on what the subjects for experimentation might look like (in terms of programming language). The main question is this: do I focus on subjects in one language (i.e., Python), or do I consider subjects across a wide variety of languages? My gut tells me to go with the latter--the benefits of this approach quickly and readily emerge: I can highlight the multi-language support of the tool, showcase novel cases for considering complexity where it hasn't been previously considered (e.g., YAMLs, Dockerfiles), and even examine the complexity of entire projects (core It'd also be worth considering doing this in two "stages"--one stage that is focused on a single language for comparison with other complexity metrics, and another that considers novel uses of I don't have any additional TODOs to report, apart from thinking about the above question and arriving at some consensus before I start in on experimental evaulation of |
Beta Was this translation helpful? Give feedback.
-
Friday, February 16Today, during the 610 afternoon classroom session, I partnered with Danny from the SE exemplar group and watched him demonstrate some of the tooling he's working on developing, as well as shared my own comp artifact. (Turns out there's some striking similarities between the two!) I offered some insights for a roadblock he was running into, and jotted them down in the GitHub issue linked here. |
Beta Was this translation helpful? Give feedback.
-
Wednesday, February 21Today in my weekly meeting with GK I bounced some additional thoughts from our previous conversation--what does my experimental results section look like? We've landed on the following agreed-upon plan:
|
Beta Was this translation helpful? Give feedback.
-
Future Research Ideas!Friday, March 1Today's meeting with GK consisted of me articulating progress with the experimental evaluation (parsing through Shortcomings for future research:
|
Beta Was this translation helpful? Give feedback.
-
Monday, March 11I met briefly with GK--the focus of this particular conversation was catching him up on what I've done/what still has to be done. The list of outstanding TODO's is as follows:
|
Beta Was this translation helpful? Give feedback.
-
Monday, March 25Today's meeting was focused on discussing questions/concerns around the thesis defense presentation. Some of the key insights GK presented to me were:
|
Beta Was this translation helpful? Give feedback.
-
|
Hello @jnormile, you have submitted a very good research notebook for the Spring 2024 semester. Thanks! |
Beta Was this translation helpful? Give feedback.
-
This space will primarily be used to track the outcomes of twice-weekly discussions with my first reader--Professor Kapfhammer--regarding the 2023-2024 SE Senior Comp. It may also contain details pertaining to outside research materials not directly discussed in the meetings otherwise outlined in this space as well as links to GitHub repositories with related tooling (or technical blog posts discussing said tools).
Beta Was this translation helpful? Give feedback.
All reactions