Project health data collection #24
Comments
I would suggest that we do not want to completely replace project reports with a completely pre-populated report. There is still value to having the maintainers complete the population of a project report as can be seen at The Importance of TSC Quarterly Project Updates. These are the things that I believe we are trying to do:
|
|
For me the big question is what do we pull from these data sources? We could generate dozens of graphs and charts from Insights and Github, although that's not going to be helpful. Some of the data available doesn't seem all that relevant (like the top emojis used in chat) and some could even be counter-productive to include in reports (for instance, tracking lines of code added seems like it could be problematic for a variety of reasons). This could be a chance to take another look at the Project Badging document that was being worked on last year. It lists a set of things TSC members thought were important to look at. For instance, what is the contributor diversity for a project, do projects have sufficient test coverage, are projects using their official channels, how responsive are projects to new contributions, etc. Not everything there could be generated in an automated report, but this could give us a set of things we'd like to see in an automated report. https://wiki.hyperledger.org/display/~shemnon/Project+Badging+Proposal And for another link to check out, I wrote an article about community metrics several years ago that summarized my experience of trying to pull together reports of community health at Mozilla. There are some thoughts and recommendations to consider in that. https://cmxhub.com/community-health-metrics-retention-diversity-maturity/ |
|
David's link to his article reminded me that I had put together the following requirements document when I originally started working on the problem of determining community health. In case that helps in any way. |
|
I see the issue as a question of "who is interested in what in the system". We have multiple ways to get the data/metrics information, but there is no flexibility to apply intelligence on that. At present, TSC & community members do analyze the available metrics information to the best of their ability. Looking at the concern here, it would help if the TSC can arrive at the set of intelligence metrics to measure project's health, figure out outliers in the available dataset. Much better if we can use BI dashboards, that would allow us to customize graphs per individual's interests. The LFX insights does provide such a feature to some extent. For instance, it calls out percentage increase in contributors count, average PR wait time in the last quarter, metrics increase/decrease on the regular contributors etc. However, having flexibility through intelligence tooling would be great. In terms of quarterly reports at Hyperledger, maintainers can continue to provide quarterly dataset available to them through one of the sources (LFX insights, the contributors report or any others for that matter). |
|
So the first order of business is defining the dimensions TSC would like to use to properly judge the health of a project. Trying to summarize the various inputs I've got so far (thanks David, Tracy, and Ry for providing the various sources of information written in the past on the topic). It looks like we believe a healthy project, regardless of the stage of the lifecycle they are in (incubating, active, graduated), should have the following attributes (note that these are different from measurements, which are concrete data to be presented in order to properly demonstrate the attributes):
Would love for folks to comment on the list above so we can all establish a common baseline. Then we can proceed to decide what types of data are needed, recognizing that not all dimensions above can necessarily be ascertained. |
|
I like the first three bullet points under community. I believe that trend data will help with visualizing. For For the code section, I am not sure how you would measure either of these items. |
|
To follow up on @tkuhrt 's point on usefulness: one thing that I think we don't necessarily do well is track where codebases are used. A lot of code in Hyperledger gets used by people who aren't connected to the project itself and don't contribute. If we had a nice way of recording where things were used (that maybe many people didn't know about) it might be a good way to measure "usefulness." |
|
I also think it might make sense to redefine "completeness" to something like "production-ready." Production readiness would seemingly capture the current definition completeness and could also be used to incorporate things that we'd like projects to do that aren't captured by the earlier criteria (e.g. security audits). What does everyone think about this? |
+1 |
|
One more idea to measure maturity is by the number of users running it in production. I know of course that this is impossible to measure completely accurately, but we as the TSC could at least give some advice to projects that they could create issues like my favorite k8s project does: metallb/metallb#5 where they ask people to disclose the fact that they are using the project. One more thing I noticed projects doing is including the names/logos of users (companies/businesses) in their readme (if the users agree to be featured of course). It makes the project readme more convincing and it could be useful information for the TSC in terms of maturity. |
|
For maturity, I think it is helpful to state where in the project lifecycle a project is and that puts the other metrics in perspective. For instance, a brand new project that has recently started incubation will very likely not have a diverse set of maintainers yet and that is understandable. A graduated project without a diverse set of maintainers though would be more cause for concern. The article I referenced called this out as something that is important to measure since not all open source projects have defined a project lifecycle, although Hyperledger's project lifecycle is well defined. One other community attribute to consider calling out is 'responsiveness'. For instance, how long does a new contributor wait for feedback on a potential contribution or question? And I like Hart's comment about usefulness being measured by where we see a project being used. We'll never know everywhere an open source project is being used, but we can make better use of the information we do have. For instance, we have a case study library that documents how certain Hyperledger projects are being used and people who are building things with projects regularly present about them to SIGs and at meetups. Including recent case studies or presentations about a project could be useful to add in reports. That could be automated too if we are consistent in tagging this content. |
|
I recently was looking at how the Eclipse Foundation presents information about their projects and I thought this would be helpful input for this conversation. In their project listings it looks like they present automated metrics for each project. You can see an example at: https://projects.eclipse.org/projects/tools.tracecompass/who This shows some diversity metrics which seem useful and it also makes it very easy to see who is leading the project and who else is involved and that seems very useful too. This data about who is involved seems like it isn't always that discoverable for our projects. I don't necessarily agree with the usefulness of having a commit activity graph though -- seeing how many commits have happened doesn't seem all that informative and doesn't seem to say much about project health. Commit counts could go down for legitimate reasons -- for instance, maybe someone who made frequent small commits was replaced by someone contributing the same amount but they prefer less frequent larger commits. The full list of Eclipse projects is at: |
|
agree with comments above that "usefulness" is an important dimension and we need to be creative about how to capture the data. I like the suggestions offered so far:
Thanks for suggesting "production-readiness" Hart. I agree it captures a wider range of concerns than completeness. I've updated the list in the main issue description. |
tkuhrt
added
task-force-proposal
Overview
This issue tracks the discussions and the work for project health indicator data.
As part of the TSC meeting on 1/6/2022, the policy around project quarterly reports was brought up. As part of the discussions, collecting data to accurately reflect a project's health came up.
A proposal to pre-populate the project quarterly report template with project health data was made. This will help project teams to not be stressed about filling out the reports because each report already comes with useful data. Equally important, this gives the TSC members a standard set of data dimensions to review in order to properly evaluate the health of each project.
Sources of Health Data
Currently, for Hyperledger project, the following sources of data are available:
Future Requirements
What can be done to allow project health data to be accurately identified and properly captured?
Stable APIs
Insights currently doesn't offer stable APIs that can be used outside of the Insights' own dashboard UI. Having stable APIs would allow the information to be embedded in other places such as TSC wiki during review meetings.
It's important to capture point-in-time snapshots for the quarterly reports, or other places where such information is used, such as the Learning Materials Development Working Group.
This requirements for snapshots can be accomplished via one of the following ways:
More data sources for Insights
Are there other data sources that can be useful to load into Insights?
Updated List (ver. 03/02/2022)
The following bullets capture the summary from the ongoing discussions.
The text was updated successfully, but these errors were encountered: