Added logic to fetch README files, documentation, commit messages, an… #2919

SahilDhillon21 · 2024-11-13T09:59:55Z

…d issue trackers from repository APIs.

Updated Project model to include fields for README content, documentation links, commit summaries, and issue tracker counts.
Implemented API calls in update_projects.py to gather and save this data to the database.

Fixed the migration file issue that I was facing in the previous PR.

I had to remove the line "Authorization": f"token {settings.GITHUB_TOKEN}" from the header as it was giving an error saying 'Unable to fetch repository - 401'. I was unsure how to deal with it so I have removed it for now.

sentry-io · 2024-11-13T10:00:04Z

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: website/management/commands/update_projects.py

Function	Unhandled Issue
`handle`	SystemExit: 1 /project/{slug}/ `Event Count:` 1

_{Did you find this useful? React with a 👍 or 👎}

SahilDhillon21 · 2024-11-13T11:47:57Z

@DonnieBLT I believe for the CodeQL test the languages should be python, javascript; instead of being 'python javascript'. Though I am unsure about how to fix this.

DonnieBLT · 2024-11-13T14:09:30Z

The reason that the token is there is because it will help with the rate limiting, you can create a token from your github profile settings, page and can you please avoid changes to white space I’m not sure what linter settings you’re using, but maybe if we can standardize them we won’t see the white space changes

SahilDhillon21 · 2024-11-13T18:50:32Z

I've added the GitHub token and fixed the formatting. Please review and let me know.

DonnieBLT

Can you please check the comments?

DonnieBLT · 2024-11-15T05:29:54Z

website/management/commands/update_projects.py

+                    )
+
+                # Set Issue Tracker URL
+                project.issue_tracker_url = f"https://github.com/{repo_name}/issues"


We can remove this since it's universal

Understood.

website/management/commands/update_projects.py

SahilDhillon21 · 2024-11-19T15:08:38Z

I have integrated a basic summary model "facebook/bart-large-cnn". However, due to the large variations in the readme files of the repositories, the summaries aren't too effective. I had thought of pre-processing the content to only pass relevant sections, but even that seems to be difficult since there is no particular structure followed. @DonnieBLT which direction should I look into to improve this? Though openai API is paid, it could do the job really well compared to the generic python models

DonnieBLT · 2024-11-19T15:09:58Z

We can use the OpenAI we already have a API key and it is set up in the code

SahilDhillon21 · 2024-11-20T13:15:53Z

I have modified the summary and label generation to use Openai's API. Meanwhile, I am working on the UI search functionality and displaying these summaries and labels.

SahilDhillon21 · 2024-11-20T21:42:19Z

Hello @DonnieBLT, here's an overview of the search functionality:
projects-search-function.webm
It filters based on the project name, a summary, and ai labels.

Looking forward to your feedback.

DonnieBLT · 2024-11-24T01:38:35Z

website/models.py

+    readme_content = models.TextField(null=True, blank=True)
+    documentation_url = models.URLField(null=True, blank=True)
+    recent_commit_messages = models.TextField(null=True, blank=True)
+    issue_tracker_url = models.URLField(null=True, blank=True)


we don't need this because it's always /issues

DonnieBLT · 2024-11-24T01:38:51Z

website/migrations/0155_project_topics_alter_project_ai_labels.py

can these migrations be merged?

DonnieBLT · 2024-11-24T01:39:11Z

website/summarization.py

+openai.api_key = os.getenv("OPENAI_API_KEY")
+
+
+def ai_summary(text, topics=None):


can you move this to utils.py please

DonnieBLT · 2024-11-24T01:40:28Z

website/templates/website/project_list.html

@@ -22,6 +22,15 @@ <h3>Projects: {{ projects.count }}</h3>
                    <i class="fas fa-plus-circle"></i> Add Project
                </button>
            </form>
+            <form id="search-form" class="search-form">


can you make sure the search in the header works and remove this?

The search bar in the header functions slightly differently. Upon entering a query and selecting a label, it transitions to another search bar specific to the selected label. This second search bar is more focused and offers only four distinct labels to refine the search (issue, domain, user, label). It doesn't allow you to search for say specific projects or orgs.

This search form filters exclusively on project's description, labels and name. Let me know if you want to remove this and and fix the header search form to include projects and organizations

Can you combine them into the one top search?

DonnieBLT · 2024-11-24T01:40:42Z

website/templates/website/project_list.html

@@ -53,130 +62,7 @@ <h3>Projects: {{ projects.count }}</h3>
                {% endfor %}
            </ul>
        {% endif %}
-        <ul class="project-list">


can we keep this in this file please

So I assume we don't want to create a separate template and have all the code here itself? Let us finalize what to do with the search function and I'll make the changes accordingly

Yes we can combine this into one template and adjust the global search to work as you have it.

DonnieBLT · 2024-11-24T01:41:34Z

website/label_generation.py

+openai.api_key = os.getenv("OPENAI_API_KEY")
+
+
+def generate_labels(readme_content, github_topics):


can you move this to utils.py too please

DonnieBLT · 2024-11-24T01:42:45Z

website/management/commands/update_projects.py

+                        project.readme_content = readme_content
+                        readme_text = markdown_to_text(readme_content)
+                        project.ai_summary = ai_summary(readme_text, project.topics)
+                        project.ai_labels = json.loads(generate_labels(readme_text, project.topics))


should we just add the labels verbatim from the topics?

I found the AI-generated labels to be more accurate and effective, but we can surely use these topics directly. I'll modify it

DonnieBLT

requesting a few changes, we're almost there

website/management/commands/update_projects.py

SahilDhillon21 · 2024-11-24T14:09:58Z

Have made most of the changes, just need a final heads-up on what to do with the search functionality as it's quite buggy and limited. Should I create a new PR to improve its working if we go that route?

DonnieBLT

A few adjustments request

DonnieBLT · 2024-11-28T04:56:59Z

website/management/commands/update_projects.py

+                    )
+
+                # Check for Documentation URL (homepage)
+                project.documentation_url = repo_data.get("homepage")


I believe we have a homepage field

DonnieBLT · 2024-11-28T04:58:15Z

website/models.py

@@ -756,6 +756,10 @@ class Project(models.Model):
    closed_issues = models.IntegerField(default=0)
    size = models.IntegerField(default=0)
    commit_count = models.IntegerField(default=0)
+    readme_content = models.TextField(null=True, blank=True)
+    documentation_url = models.URLField(null=True, blank=True)


Use homepage_url

DonnieBLT · 2024-11-28T05:00:04Z

website/templates/website/project_list.html

@@ -22,6 +22,15 @@ <h3>Projects: {{ projects.count }}</h3>
                    <i class="fas fa-plus-circle"></i> Add Project
                </button>
            </form>
+            <form id="search-form" class="search-form">


Can you combine them into the one top search?

DonnieBLT · 2024-11-28T05:00:56Z

website/templates/website/project_list.html

@@ -53,130 +62,7 @@ <h3>Projects: {{ projects.count }}</h3>
                {% endfor %}
            </ul>
        {% endif %}
-        <ul class="project-list">


Yes we can combine this into one template and adjust the global search to work as you have it.

SahilDhillon21 · 2024-11-30T11:23:45Z

I have combined the search bar into one and added all the remaining categories. I'll raise a new pr later to improve the UI of the search results, have kept it basic for now.

SahilDhillon21 · 2024-12-02T17:02:00Z

I have merged the migrations, please check

DonnieBLT · 2024-12-14T00:22:14Z

Can you please clean this up?

SahilDhillon21 · 2024-12-14T13:10:52Z

I have started from the main branch again. Here a normal summary is used

The search bar changes are not included here, should I raise a separate pr for that or have those changes here itself?

P.S. I'm not sure how to solve this error for ai_summary:
Error generating summary:

You tried to access openai.ChatCompletion, but this is no longer supported in openai>=1.0.0 - see the README at https://github.com/openai/openai-python for the API.

You can run openai migrate to automatically upgrade your codebase to use the 1.0.0 interface.

Alternatively, you can pin your installation to the old version, e.g. pip install openai==0.28

A detailed migration guide is available here: openai/openai-python#742

Got the error even after updating openai. It could be because I don't have access to a valid api key, please verify

DonnieBLT · 2024-12-19T14:46:13Z

The OpenAI error is happening with a valid key for the pr analysis too in production. I think maybe updating and using a different function will fix it

SahilDhillon21 · 2024-12-19T14:51:11Z

Understood, will resolve it

SahilDhillon21 · 2024-12-19T16:03:33Z

The functions are migrated
Please run pip install --upgrade openai

As for the test that's failing, it seems to be working when I set a random API key on my local machine, so hopefully it works in production as well
It updates the functions used in pr_analysis as well

SahilDhillon21 force-pushed the ai-project-enhancement branch from 95dc670 to 34add4f Compare November 13, 2024 17:32

DonnieBLT requested changes Nov 19, 2024

View reviewed changes

DonnieBLT reviewed Nov 24, 2024

View reviewed changes

DonnieBLT requested changes Nov 24, 2024

View reviewed changes

DonnieBLT added the requested changes label Nov 24, 2024

DonnieBLT reviewed Nov 24, 2024

View reviewed changes

website/management/commands/update_projects.py Outdated Show resolved Hide resolved

DonnieBLT requested changes Nov 28, 2024

View reviewed changes

cleanup changes

fd1dbc3

SahilDhillon21 force-pushed the ai-project-enhancement branch from 26ef4fe to fd1dbc3 Compare December 14, 2024 12:56

remove bold from summary

bfaaeab

DonnieBLT previously approved these changes Dec 18, 2024

View reviewed changes

DonnieBLT and others added 2 commits December 18, 2024 05:43

Merge branch 'main' into ai-project-enhancement

30cfecb

merge migrations

8fa0c22

SahilDhillon21 dismissed DonnieBLT’s stale review via 8fa0c22 December 19, 2024 12:59

SahilDhillon21 and others added 4 commits December 19, 2024 18:29

Merge branch 'main' into ai-project-enhancement

41f2406

merge

7fb72c4

Merge branch 'main' into ai-project-enhancement

c49c30d

fix conflicts

9998002

migrate openai code to latest version

8dcc1c6

Merge branch 'main' into ai-project-enhancement

f2e70ab

		openai.api_key = os.getenv("OPENAI_API_KEY")


		def ai_summary(text, topics=None):

		openai.api_key = os.getenv("OPENAI_API_KEY")


		def generate_labels(readme_content, github_topics):

Added logic to fetch README files, documentation, commit messages, an… #2919

Are you sure you want to change the base?

Added logic to fetch README files, documentation, commit messages, an… #2919

Conversation

SahilDhillon21 commented Nov 13, 2024 • edited Loading

sentry-io bot commented Nov 13, 2024

🔍 Existing Issues For Review

SahilDhillon21 commented Nov 13, 2024

DonnieBLT commented Nov 13, 2024

SahilDhillon21 commented Nov 13, 2024

DonnieBLT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 19, 2024

DonnieBLT commented Nov 19, 2024

SahilDhillon21 commented Nov 20, 2024

SahilDhillon21 commented Nov 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DonnieBLT left a comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 24, 2024

DonnieBLT left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SahilDhillon21 commented Nov 30, 2024

SahilDhillon21 commented Dec 2, 2024

DonnieBLT commented Dec 14, 2024

SahilDhillon21 commented Dec 14, 2024 • edited Loading

DonnieBLT commented Dec 19, 2024

SahilDhillon21 commented Dec 19, 2024

SahilDhillon21 commented Dec 19, 2024

SahilDhillon21 commented Nov 13, 2024 •

edited

Loading

SahilDhillon21 commented Dec 14, 2024 •

edited

Loading