Home
Problem
Status
Contest
Workbook
User
Group
Forum
Register
Login
{"managingGroups":{},"author":"mostafa_galal1","updateTime":1697029164000,"title":"Scraping Engine for Competitive Programming Accelerated Retriever (Secpar)","dislikeCnt":0,"content":"# Scraping Engine for Competitive Programming Accelerated Retriever (Secpar)\n\n\u003cbr\u003e\n\n## Overview\n\nSecpar is a Python command-line tool designed to scrape code submissions from various online programming platforms and store them in a GitHub repository. It supports platforms such as Codeforces, CSES (University of Helsinki), and Vjudge. This documentation provides a detailed overview of the Scraper\u0027s functionalities and how to use them.\n\n##### Demo Repo: [CP-Submissions](https://github.com/MostafaGalal1/CP-Submissions)\n\n\u003cbr\u003e\n\n## Table of Contents\n\n1. [Features](#features)\n2. [Installation](#installation)\n3. [Usage](#usage)\n - [Initialization](#initialization)\n - [Scraping](#scraping)\n4. [Command-Line Interface](#command-line-interface)\n5. [Scraper Configuration](#scraper-configuration)\n6. [Customization](#customization)\n7. [Data Storage](#data-storage)\n8. [FAQs](#faqs)\n9. [Contributing](#contributing)\n10. [License](#license)\n11. [Upcoming Features](#upcoming-features) \n\u003cbr\u003e\n## 1. Features \u003ca name\u003d\"features\"\u003e\u003c/a\u003e\n\n- **Supported Platforms**: Secpar supports the following programming platforms:\n - Codeforces\n - CSES (University of Helsinki)\n - Vjudge\n\n- **GitHub Integration**: Code submissions are stored in a GitHub repository, making it easy to manage and share your solutions.\n\n- **Automatic README Generation**: Secpar automatically generates a README file for your GitHub repository, listing all your code submissions with problem details, links, and tags.\n\n- **Incremental Scraping**: Secpar keeps track of previously scraped submissions, ensuring that only new submissions are added to your repository.\n \n- **Multi-accounts Scraping**: Scrape the same platform more than once in case of having multiple accounts on that platform without worry of redundancy.\n\n- **Authentication**: Securely authenticate with the supported platforms to access your submissions.\n\n- **Customization**: Customize the formatting of your README and configure other scraper options.\n\u003cbr\u003e\n## 2. Installation \u003ca name\u003d\"installation\"\u003e\u003c/a\u003e\n\nTo use Secpar, follow these installation steps:\n\n**Install Secpar**: Install Secpar package on your local machine:\n\n```shell\npip install Secpar\n```\n\u003cbr\u003e\n## 3. Usage \u003ca name\u003d\"usage\"\u003e\u003c/a\u003e\n\nSecpar has two primary modes of operation: initialization and scraping.\n\n### Initialization \u003ca name\u003d\"initialization\"\u003e\u003c/a\u003e\n\nInitialization is the first step to configure your scraper for a GitHub repository.\n\n1. Run the initialization command:\n\n ```shell\n python main.py -c init\n ```\n\n2. Follow the prompts to enter your GitHub username, repository name, and access token.\n\n### Scraping \u003ca name\u003d\"scraping\"\u003e\u003c/a\u003e\n\nScraping allows you to retrieve code submissions from supported platforms and store them in your GitHub repository.\n\n1. To scrape submissions, use the following command:\n\n ```shell\n Secpar -s PLATFORM_NAME\n ```\n\n Replace `PLATFORM_NAME` with one of the supported platforms: `codeforces`, `cses`, or `vjudge`.\n\n2. Depending on the platform, you may need to provide additional information such as your platform password.\n\n3. Secpar will fetch new submissions and update your GitHub repository.\n\n### Note:\nTo upload submissions codes for codeforces you need to have tor installed and inside your torrc file place these two line:\n\n```shell\nSocksPort 9050\nControlPort 9051\n```\n\nOpen tor tab and make sure you can browse using it and keep it open till you scrape using the terminal.\n\u003cbr\u003e\n## 4. Command-Line Interface \u003ca name\u003d\"command-line-interface\"\u003e\u003c/a\u003e\n\nSecpar provides a command-line interface with the following options:\n\n- `-c`, `--command`: Specify the command (`init` for initialization or `update` for scraping).\n\n- `-s`, `--scrape`: Specify the platform to scrape data from (`codeforces`, `cses`, or `vjudge`).\n\n- `-h`, `--help`: Display usage instructions and available options.\n\nExample usage:\n\n```shell\npython main.py -c init\npython main.py -s codeforces\n```\n\u003cbr\u003e\n## 5. Scraper Configuration \u003ca name\u003d\"scraper-configuration\"\u003e\u003c/a\u003e\n\nSecpar can be configured in several ways:\n\n- **GitHub Configuration**: Set up your GitHub repository details and access token during initialization.\n\n- **Platform Authentication**: Authenticate with your platform credentials (e.g., Codeforces username and password) for scraping.\n\n- **Customization**: Customize the formatting of your README and configure scraper options in the code (e.g., maximum requests, submission per update).\n\u003cbr\u003e\n## 6. Customization \u003ca name\u003d\"customization\"\u003e\u003c/a\u003e\n\nYou can customize Secpar\u0027s behavior by modifying the source code. Here are some customization options:\n\n- **Formatting**: Customize the formatting of the generated README for each platform. You can modify the formatting in the corresponding `Formatter` class.\n\n- **Configuration**: Adjust Secpar\u0027s settings, such as maximum requests, submissions per update, or other platform-specific parameters.\n\u003cbr\u003e\n## 7. Data Storage \u003ca name\u003d\"data-storage\"\u003e\u003c/a\u003e\n\nSecpar stores code submissions and related information in your GitHub repository. Each submission is listed in the README with details such as problem name, language, solution link, tags, and submission date.\n\u003cbr\u003e\n## 8. FAQs \u003ca name\u003d\"faqs\"\u003e\u003c/a\u003e\n\n- **What platforms does Secpar support?**\n - Secpar currently supports Codeforces, CSES (University of Helsinki), and Vjudge.\n\n- **Is it safe to store my GitHub access token?**\n - Access tokens should be stored securely. Secpar stores them in a configuration file, and it\u0027s essential to protect this file.\n\n- **How can I customize the README format?**\n - You can customize the README format by modifying the corresponding `Formatter` class for each platform.\n\u003cbr\u003e\n## 9. Contributing \u003ca name\u003d\"contributing\"\u003e\u003c/a\u003e\n\nContributions to Secpar are welcome! Feel free to fork the repository, make improvements, and create pull requests.\n\u003cbr\u003e\n## 10. License \u003ca name\u003d\"license\"\u003e\u003c/a\u003e\n\nSecpar is released under the MIT License. See the [LICENSE](https://github.com/MostafaGalal1/Secpar/blob/main/LICENSE) file for details.\n\u003cbr\u003e\n## 11. Upcoming Features \u003ca name\u003d\"upcoming-features\"\u003e\n- More platforms: platforms such as Atcoder and CodeChef are currently being worked on.\n\n---\n\n*Note: This documentation provides an overview of Secpar\u0027s functionality and usage. For detailed code explanations, refer to the source code and comments in Secpar\u0027s repository.*\n","threadId":172633,"likeCnt":1,"createTime":1697029164000,"isWorkbook":false,"viewCnt":175,"openness":2,"fav":false,"id":4177,"trustable":false}