Store a list of current repos which use the JSON Schema topic and when they were created #4 #10

wcj617 · 2024-03-22T04:49:30Z

Resolves #4

This PR is for 'Store a list of current repos which use the JSON Schema topic and when they were created
#4'

Requirement

Here's the work that needs to be done on this:

 No longer check the way back machine for earliest use of the json-schema github topic and no longer record the data related to the way back machine
 Get the date of the first commit for a repo and store that in the CSV as a timestamp
 Confirm the initial data script can be run fully and include the created CSV as a commit
 Catch any potential error and log into a file, without causing the script to crash, and allowing it to continue if appropriate.
Stretch objectives:

 
![2024-03-27_03-36-48](https://github.com/json-schema-org/ecosystem/assets/47830372/35f647fe-ca4c-46d7-8c09-f36c2ed5ce87)
Write tests which [mock the Github API calls](https://mswjs.io/) and [mock the file writing](https://medium.com/nerd-for-tech/testing-in-node-js-easy-way-to-mock-filesystem-883b9f822ea4), and check it has the correct content for the API data mocked.

What I have done

This commit includes several enhancements to the initial data script:

The script now retrieves the date of the first commit for each repository and stores it in the CSV file as a timestamp.
The script has been confirmed to run fully, and the resulting CSV file is included in this commit.
Error handling has been added to the script. Any errors that occur are logged to a file, and the script is able to continue running where appropriate.

… tests

Relequestual

Welcome, and thanks for your initial work on this.
I have left a few comments and questions.

Looking at the requirements on #4.

Covered.
I have updated this to specify using the timestamp format.
Not covered. "Fully run" means all results without limit. There are about 1.8k results. This may take some time to run.
Not covered. This is important because it can take some time to run, so the script should capture errors and continue running rather than crash and exit.

When you have updated and tested, please indicate this needs to be re-reviewed. Thanks.
Feel free to reach me on our Slack under the same username if you have questions.

projects/initial-data/.gitignore

projects/initial-data/package.json

projects/initial-data/main.js

…data

deleted unnecessary file

wcj617 · 2024-03-30T23:03:10Z

This is my second qualification task work!
second qualification task.pdf

Relequestual · 2024-04-04T08:35:43Z

This looks much improved. I'm scheduling some time to test it locally before it is merged. Thanks for your work on this.

… tests

…data

deleted unnecessary file

Relequestual · 2024-05-14T09:43:30Z

I rebased and force pushed to this PR.
Rebase was required because importing a function from main.js would cause the main function to be called, and an API request to Github to be maid before the mock could be setup, which resulted in seeing a failed API request (due to invalid API key as it had expired) after all tests had run. This was confusing when all tests had passed.

Relequestual

Left a few suggestions, otherwise this is looking better.

I know GSoC candidates have been selected now, but it would be nice to know if you'd like to finish this PR or not. If not, that's totally fine, and I'm sure I can find someone else to finish it or do it myself. Let me know.

projects/initial-data/main.js

Relequestual · 2024-05-14T09:49:34Z

projects/initial-data/main.js

+      return Date.parse(lastPageResponse.data[0].commit.author.date);
+    } else {
+      console.error('Error occured');
+      throw new Error('No commits found'); //TODO: check if this is the correct error message


Suggested change

throw new Error('No commits found'); //TODO: check if this is the correct error message

throw new Error(`No commits found ${owner}/${repo}`);

fixed as suggested

Relequestual · 2024-05-14T09:51:12Z

projects/initial-data/main.js

+    }
+  } catch (err) {
+    console.log(err);
+    throw new Error('No commits found'); //TODO: check if this is the correct error message


Suggested change

throw new Error('No commits found'); //TODO: check if this is the correct error message

throw new Error(`Could not find any commits for ${owner}/${repo}`);

fixed as suggested

Relequestual · 2024-05-14T09:52:21Z

projects/initial-data/main.js

+      if (response.data.length > 0) {
+        response.data[0].created_at;
+      } else {
+        throw new Error('No releases found'); //TODO: check if this is the correct error message


Suggested change

throw new Error('No releases found'); //TODO: check if this is the correct error message

throw new Error(`No releases found for ${owner}/${repo}`);

fixed as suggested

Relequestual · 2024-05-14T09:52:41Z

projects/initial-data/main.js

+    if (lastPageResponse.data.length > 0) {
+      return Date.parse(lastPageResponse.data[0].created_at);
+    } else {
+      throw new Error('No releases found'); //TODO: check if this is the correct error message


Suggested change

throw new Error('No releases found'); //TODO: check if this is the correct error message

throw new Error(`No releases found for ${owner}/${repo}`);

fixed as suggested

Relequestual · 2024-05-14T09:53:16Z

projects/initial-data/main.js

+    }
+  } catch (err) {
+    console.error('Error occured');
+    throw new Error('No releases found'); //TODO: check if this is the correct error message


Suggested change

throw new Error('No releases found'); //TODO: check if this is the correct error message

throw new Error(`Unable to get releases for ${owner}/${repo}`);

Relequestual · 2024-05-14T09:54:48Z

projects/initial-data/main.js

+    firstReleaseDate = await fetchFirstReleaseDate(octokit, owner, repo);
+  } catch (err) {
+    console.error('Error occured in fetchFirstReleaseDate');
+    throw new Error('No releases found');


Suggested change

throw new Error('No releases found');

throw new Error(`Unable to get releases for ${owner}/${repo}`);

Relequestual · 2024-05-14T09:55:59Z

projects/initial-data/main.js

+  try {
+    firstCommitDate = await fetchFirstCommitDate(octokit, owner, repo);
+  } catch (err) {
+    console.error('Error occured in fetchFirstCommitDate');


Suggested change

console.error('Error occured in fetchFirstCommitDate');

console.error(`Error occured in fetchFirstCommitDate for ${owner}/${repo}`);

Relequestual · 2024-05-14T09:56:52Z

projects/initial-data/main.js

+    firstCommitDate = await fetchFirstCommitDate(octokit, owner, repo);
+  } catch (err) {
+    console.error('Error occured in fetchFirstCommitDate');
+    throw new Error('No commits found');


Suggested change

throw new Error('No commits found');

throw new Error(`Error trying to find first commit for ${owner}/${repo}`);

wcj617 · 2024-05-24T05:36:59Z

[previous git commit history]

[after fixed]

[mock jest]

Hi Ben,

Thank you for your insightful feedback! I appreciate the guidance and the opportunity to learn from real-world problems provided by this project.

I've completed a git rebase and resolved the divergent commits as you've noticed. I've also re-requested a review of the PR to ensure everything aligns with the project's current needs.

Regarding the API key update, your clarification was very helpful. I understand now the importance of designing tests that don't make real API calls and will take this as a key learning point.

Thank you for your support and for providing a nurturing environment for newcomers like me. I look forward to continuing my contribution to the project and will keep in touch via Slack as needed.

Best regards,
David

chore: Initiate GSoC qualification task, fix package.json and add CSV…

b86c960

… tests

wcj617 changed the title ~~chore: Initiate GSoC qualification task, fix package.json and add CSV…~~ chore: Initiate GSoC qualification task, fix package.json and add CSV#4 Mar 22, 2024

wcj617 added 2 commits March 23, 2024 22:14

feat#4: Retrieve and store date of first commit in CSV

d37ad55

refactor: Remove Wayback Machine checks and related data recording

339d98c

Relequestual changed the title ~~chore: Initiate GSoC qualification task, fix package.json and add CSV#4~~ chore: Initiate GSoC qualification task, fix package.json and add CSV Mar 25, 2024

Relequestual requested changes Mar 25, 2024

View reviewed changes

projects/initial-data/.gitignore Outdated Show resolved Hide resolved

projects/initial-data/package.json Show resolved Hide resolved

projects/initial-data/main.js Outdated Show resolved Hide resolved

projects/initial-data/main.js Outdated Show resolved Hide resolved

Relequestual mentioned this pull request Mar 25, 2024

Track the number of stars and forks from repos that use the json-schema topic, over time #2

Open

wcj617 added 2 commits March 25, 2024 14:19

chore: Re-add .gitignore file

8dde501

fix: Correct eslint:fix script in package.json

8a46389

wcj617 changed the title ~~chore: Initiate GSoC qualification task, fix package.json and add CSV~~ PR for Store a list of current repos which use the JSON Schema topic and when they were created #4 Mar 26, 2024

wcj617 added 5 commits March 27, 2024 02:05

test: Add integration tests for Github API calls and file writing

f96535c

feat: Enhance initial data script with error handling and additional …

483a164

…data

Delete projects/initial-data/initialTopicRepoData-1711072195857.csv

3f9c9fb

deleted unnecessary file

Delete projects/initial-data/initialTopicRepoData-1711256938624.csv

584c825

Delete projects/initial-data/initialTopicRepoData-1711263053033.csv

0786fde

wcj617 requested a review from Relequestual March 27, 2024 10:28

Fix integration test: Resolve Unix timestamp conversion error

6761a8e

wcj617 changed the title ~~PR for Store a list of current repos which use the JSON Schema topic and when they were created #4~~ Store a list of current repos which use the JSON Schema topic and when they were created #4 Apr 2, 2024

wcj617 added 10 commits May 14, 2024 10:35

chore: Initiate GSoC qualification task, fix package.json and add CSV…

543ebf0

… tests

feat#4: Retrieve and store date of first commit in CSV

2849433

refactor: Remove Wayback Machine checks and related data recording

a2ca6ef

chore: Re-add .gitignore file

cbbe873

fix: Correct eslint:fix script in package.json

4f64dfd

test: Add integration tests for Github API calls and file writing

30a8a8c

feat: Enhance initial data script with error handling and additional …

af19955

…data

Delete projects/initial-data/initialTopicRepoData-1711072195857.csv

9e63001

deleted unnecessary file

Delete projects/initial-data/initialTopicRepoData-1711256938624.csv

860921a

Delete projects/initial-data/initialTopicRepoData-1711263053033.csv

0e0dd5b

Fix integration test: Resolve Unix timestamp conversion error

b5e2c96

Relequestual force-pushed the main branch from 6761a8e to b5e2c96 Compare May 14, 2024 09:41

Relequestual requested changes May 14, 2024

View reviewed changes

This was referenced May 14, 2024

feat: #5. Add first commit date to initial repository data #9

Closed

Create fetchRepos.js #8

Closed

Get repo topics and created at date data from search results #11

Open

wcj617 added 2 commits May 23, 2024 22:11

Fix improper error handling

faecfd7

Merge remote-tracking branch 'origin'

3dd0683

wcj617 requested a review from Relequestual May 24, 2024 05:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store a list of current repos which use the JSON Schema topic and when they were created #4 #10

Store a list of current repos which use the JSON Schema topic and when they were created #4 #10

wcj617 commented Mar 22, 2024 •

edited

Relequestual left a comment

wcj617 commented Mar 30, 2024

Relequestual commented Apr 4, 2024

Relequestual commented May 14, 2024

Relequestual left a comment

Relequestual May 14, 2024

wcj617 May 17, 2024

Relequestual May 14, 2024

wcj617 May 17, 2024

Relequestual May 14, 2024

wcj617 May 17, 2024

Relequestual May 14, 2024

wcj617 May 17, 2024

Relequestual May 14, 2024

Relequestual May 14, 2024

Relequestual May 14, 2024

Relequestual May 14, 2024

wcj617 commented May 24, 2024 •

edited

	throw new Error('No commits found'); //TODO: check if this is the correct error message
	throw new Error(`No commits found ${owner}/${repo}`);

	throw new Error('No commits found'); //TODO: check if this is the correct error message
	throw new Error(`Could not find any commits for ${owner}/${repo}`);

	throw new Error('No releases found'); //TODO: check if this is the correct error message
	throw new Error(`No releases found for ${owner}/${repo}`);

	throw new Error('No releases found'); //TODO: check if this is the correct error message
	throw new Error(`Unable to get releases for ${owner}/${repo}`);

	throw new Error('No releases found');
	throw new Error(`Unable to get releases for ${owner}/${repo}`);

	console.error('Error occured in fetchFirstCommitDate');
	console.error(`Error occured in fetchFirstCommitDate for ${owner}/${repo}`);

	throw new Error('No commits found');
	throw new Error(`Error trying to find first commit for ${owner}/${repo}`);

Store a list of current repos which use the JSON Schema topic and when they were created #4 #10

Are you sure you want to change the base?

Store a list of current repos which use the JSON Schema topic and when they were created #4 #10

Conversation

wcj617 commented Mar 22, 2024 • edited

Requirement

What I have done

Relequestual left a comment

Choose a reason for hiding this comment

wcj617 commented Mar 30, 2024

Relequestual commented Apr 4, 2024

Relequestual commented May 14, 2024

Relequestual left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wcj617 commented May 24, 2024 • edited

wcj617 commented Mar 22, 2024 •

edited

wcj617 commented May 24, 2024 •

edited