Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merging develop branch #6

Merged
merged 7 commits into from Jul 5, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion .npmignore
@@ -1,2 +1,3 @@
.github
node_modules
node_modules
src/
2 changes: 1 addition & 1 deletion AVAILABLE_LANGUAGES.md
@@ -1,7 +1,7 @@
# Here is available language you can use in this library

- Indonesia ~ If your hosting location or application location is in Indonesia, you can enter code `id`
- United States ~ If your hosting location or application location is in America or United States (US), you can enter code `us`.
- ~~United States ~ If your hosting location or application location is in America or United States (US), you can enter code `us`.~~ Removed.
- Spain ~ If your hosting location or application location is in Spain, you can enter code `es`.
- Portuguese ~ If your hosting location or application location is in Portuguese, you can enter code `pt`.
- Russia ~ If your hosting location or application location is in Russia, you can enter code `ru`.
Expand Down
11 changes: 9 additions & 2 deletions README.md
@@ -1,9 +1,9 @@
# Brainly Scraper V2
This library retrieves data from Brainly that has been designed to avoid `403 Forbidden` exception.

> To avoid such errors, you can fill in a valid country code according to the location of your hosting server. For example, heroku, heroku with the United States region. That is, you must fill in the code `us` as the country code inside the brainly parameter constructor.
> To avoid such errors, you can fill in a valid country code. You can test the 10 languages or country codes available to see if your server hosting country location or location is rejected.

You can test the 10 languages or country codes available to see if your server hosting country location or location is rejected. See https://github.com/hansputera/brainly-scraper-languages/blob/master/AVAILABLE_LANGUAGES.md
See https://github.com/hansputera/brainly-scraper-languages/blob/master/AVAILABLE_LANGUAGES.md

# 💉 Installation
- Using NPM : `npm install brainly-scraper-v2`
Expand Down Expand Up @@ -52,5 +52,12 @@ brain.search("es", "Pythagoras").then(console.log).catch(console.error);
# ⚙️ Issues and Bugs
If you have problems using this library, you can create an issue in the [github repository](https://github.com/hansputera/brainly-scraper-languages). Remember, don't forget to read the instructions and try.

# 🔬 Hosting testing
## Replit.com
Free and paid hosting replit already tested. The country codes that pass are Indonesia, Spain, India, Portuguese, and Philipines.


Ever tried and tested on other hosting? Please send us your feedback in PR. That would be very helpful 😊.

# ✍️ Contributions
Do you want to contribute with this library for the better? Very well, fork this [github repository](https://github.com/hansputera/brainly-scraper-languages) then install dependencies to your directory. Happy coding 😁
1 change: 0 additions & 1 deletion dist/src/config.d.ts
@@ -1,7 +1,6 @@
export declare const graphql_query = "query SearchQuery($query: String!, $first: Int!, $after: ID) {\n questionSearch(query: $query, first: $first, after: $after) {\n count\n edges {\n node {\n databaseId\n content\n points\n created\n lastActivity\n subject {\n name\n slug\n }\n grade {\n name\n slug\n }\n attachments {\n url\n }\n author {\n databaseId\n nick\n points\n gender\n description\n isDeleted\n avatar {\n url\n }\n category\n clientType\n rank {\n databaseId\n name\n }\n receivedThanks\n bestAnswersCount\n helpedUsersCount\n }\n isAuthorsFirstQuestion\n canBeAnswered\n pointsForAnswer\n pointsForBestAnswer\n answers {\n nodes {\n databaseId\n content\n points\n isBest\n created\n rating\n ratesCount\n thanksCount\n attachments {\n url\n }\n author {\n databaseId\n nick\n points\n gender\n description\n isDeleted\n avatar {\n url\n }\n category\n clientType\n rank {\n databaseId\n name\n }\n receivedThanks\n bestAnswersCount\n helpedUsersCount\n }\n }\n }\n }\n }\n }\n }";
export declare const baseURLs: {
id: string;
us: string;
es: string;
pt: string;
ru: string;
Expand Down
2 changes: 1 addition & 1 deletion dist/src/config.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions dist/src/main.d.ts
@@ -1,12 +1,12 @@
import type { Answer, LanguageList, Question } from "./types";
import type { Answer, CountryList, LanguageList, Question } from "./types";
export default class Brainly {
country: LanguageList;
country: CountryList;
clientRequest: (lang: LanguageList) => import("got").Got;
/**
*
* @param country - Here, please put your application server country code, if your server are in United States. Enter region/country code `us` to this parameter. Because what? All brainly website is protected, if you do not enter valid region/country code. It will trigger an Error Exception.
* @param country - Here, please put your application server country code. if you do not enter valid region/country code. It will trigger an Error Exception.
*/
constructor(country?: LanguageList);
constructor(country?: CountryList);
/**
* Use this function if you want search question, it will returns question detail, question author, answer detail, attachments (if question or answer attachments is any), rating question and answer.
*
Expand Down
152 changes: 81 additions & 71 deletions dist/src/main.js
Expand Up @@ -9,19 +9,24 @@ const error_1 = __importDefault(require("./error"));
const random_useragent_1 = __importDefault(require("random-useragent"));
const util_1 = __importDefault(require("./util"));
class Brainly {
country;
clientRequest = (lang) => got_1.default.extend({
prefixUrl: `${this.getBaseURL(lang)}/graphql`,
headers: {
"user-agent": this.getAgent()
}
});
/**
*
* @param country - Here, please put your application server country code, if your server are in United States. Enter region/country code `us` to this parameter. Because what? All brainly website is protected, if you do not enter valid region/country code. It will trigger an Error Exception.
* @param country - Here, please put your application server country code. if you do not enter valid region/country code. It will trigger an Error Exception.
*/
constructor(country = "id") {
this.country = country;
this.clientRequest = (lang) => got_1.default.extend({
prefixUrl: `${this.getBaseURL(this.country)}/graphql`,
headers: {
"user-agent": this.getAgent(),
"origin": this.getBaseURL(lang),
"sec-gpc": "1",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"batch": "true"
}
});
if (!this.isValidLanguage(country))
throw new error_1.default("Please put valid country!");
}
Expand All @@ -33,69 +38,74 @@ class Brainly {
* @param length Length array from question list
*/
async search(language = "id", question, length = 10) {
if (!this.isValidLanguage(language))
throw new error_1.default("Please put valid language!");
const body = this.getRequestBody(question, length);
const response = await this.clientRequest(this.country.toLowerCase()).post(language.toLowerCase(), {
json: body
});
const validJSON = JSON.parse(response.body)[0].data.questionSearch.edges;
const objects = validJSON.map(obj => {
const question = {
id: obj.node.databaseId,
content: util_1.default.clearContent(obj.node.content),
attachments: obj.node.attachments.map(attach => attach.url),
author: obj.node.author ? {
id: obj.node.author.databaseId,
avatar_url: obj.node.author.avatar ? obj.node.author.avatar.url : undefined,
deleted: obj.node.author.isDeleted,
url: `${this.getBaseURL(language)}/app/profile/${obj.node.author.databaseId}`,
rank: obj.node.author.rank.name,
username: obj.node.author.nick,
receivedThanks: obj.node.author.receivedThanks,
bestAnswersCount: obj.node.author.bestAnswersCount,
gender: obj.node.author.gender,
description: obj.node.author.description,
points: obj.node.author.points,
helpedUsersCount: obj.node.author.helpedUsersCount
} : undefined,
points: {
points: obj.node.points,
forBest: obj.node.pointsForBestAnswer
},
grade: obj.node.grade.name,
education: obj.node.subject.name,
created: obj.node.created,
can_be_answered: obj.node.canBeAnswered,
url: `${this.getBaseURL(language)}/${util_1.default.resolveWorkName(language.toLowerCase())}/${obj.node.databaseId}`
};
const answers = obj.node.answers.nodes.map(answerObj => ({
content: util_1.default.clearContent(answerObj.content),
attachments: answerObj.attachments.map(attach => attach.url),
rates: answerObj.ratesCount,
rating: answerObj.rating,
isBest: answerObj.isBest,
created: answerObj.created,
author: answerObj.author ? {
id: answerObj.author.databaseId,
username: answerObj.author.nick,
gender: answerObj.author.gender,
avatar_url: answerObj.author.avatar ? answerObj.author.avatar.url : undefined,
deleted: answerObj.author.isDeleted,
url: `${this.getBaseURL(language.toLowerCase())}/app/profile/${answerObj.author.databaseId}`,
description: answerObj.author.description,
bestAnswersCount: answerObj.author.bestAnswersCount,
points: answerObj.author.points,
helpedUsersCount: answerObj.author.helpedUsersCount,
receivedThanks: answerObj.author.receivedThanks,
rank: answerObj.author.rank.name
} : undefined
}));
return {
question, answers
};
});
return objects;
try {
if (!this.isValidLanguage(language))
throw new error_1.default("Please put valid language!");
const body = this.getRequestBody(question, length);
const response = await this.clientRequest(this.country.toLowerCase()).post(language.toLowerCase(), {
json: body
});
const validJSON = JSON.parse(response.body)[0].data.questionSearch.edges;
const objects = validJSON.map(obj => {
const question = {
id: obj.node.databaseId,
content: util_1.default.clearContent(obj.node.content),
attachments: obj.node.attachments.map(attach => attach.url),
author: obj.node.author ? {
id: obj.node.author.databaseId,
avatar_url: obj.node.author.avatar ? obj.node.author.avatar.url : undefined,
deleted: obj.node.author.isDeleted,
url: `${this.getBaseURL(language)}/app/profile/${obj.node.author.databaseId}`,
rank: obj.node.author.rank ? obj.node.author.rank.name : "-",
username: obj.node.author.nick,
receivedThanks: obj.node.author.receivedThanks,
bestAnswersCount: obj.node.author.bestAnswersCount,
gender: obj.node.author.gender,
description: obj.node.author.description,
points: obj.node.author.points,
helpedUsersCount: obj.node.author.helpedUsersCount
} : undefined,
points: {
points: obj.node.points,
forBest: obj.node.pointsForBestAnswer
},
grade: obj.node.grade.name,
education: obj.node.subject.name,
created: obj.node.created,
can_be_answered: obj.node.canBeAnswered,
url: `${this.getBaseURL(language)}/${util_1.default.resolveWorkName(language.toLowerCase())}/${obj.node.databaseId}`
};
const answers = obj.node.answers.nodes.map(answerObj => ({
content: util_1.default.clearContent(answerObj.content),
attachments: answerObj.attachments.map(attach => attach.url),
rates: answerObj.ratesCount,
rating: answerObj.rating,
isBest: answerObj.isBest,
created: answerObj.created,
author: answerObj.author ? {
id: answerObj.author.databaseId,
username: answerObj.author.nick,
gender: answerObj.author.gender,
avatar_url: answerObj.author.avatar ? answerObj.author.avatar.url : undefined,
deleted: answerObj.author.isDeleted,
url: `${this.getBaseURL(language.toLowerCase())}/app/profile/${answerObj.author.databaseId}`,
description: answerObj.author.description,
bestAnswersCount: answerObj.author.bestAnswersCount,
points: answerObj.author.points,
helpedUsersCount: answerObj.author.helpedUsersCount,
receivedThanks: answerObj.author.receivedThanks,
rank: answerObj.author.rank ? answerObj.author.rank.name : "-"
} : undefined
}));
return {
question, answers
};
});
return objects;
}
catch (err) {
throw new Error(JSON.stringify(err));
}
}
getRequestBody(question, length = 10) {
return [{
Expand Down
4 changes: 2 additions & 2 deletions dist/src/tests/index.test.js
Expand Up @@ -5,8 +5,8 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
Object.defineProperty(exports, "__esModule", { value: true });
const index_1 = __importDefault(require("../../index"));
it("should return information about question and answer", (done) => {
const brain = new index_1.default("id");
brain.search("ru", "Pythagoras").then((results) => {
const brain = new index_1.default("hi");
brain.search("us", "Pythagoras").then((results) => {
console.log(results[0].question);
expect(results).toBeDefined();
done();
Expand Down