Downloader Core

Download DODFs from the Buriti Website and save on proper directory.

Download monthly pdfs of DODFs.

Usage example:

downloader = Downloader()
downloader.pull(start_date, end_date)

Downloader Class

class dodfminer.downloader.core.Downloader(save_path='./')[source]

Responsible for the download of the DODFs Pdfs.

Parameters

save_path (str) – Path to save the downloads.

_download_path

Folder in which the downloads will be stored.

_prog_bar

Indicate if download should contain a progress bar.

pull(start_date, end_date)[source]

Make the download of the DODFs pdfs.

All dodfs are downloaded from start_date to end_date inclusively. The Pdfs are saved in a folder called “data” inside the project folder.

Parameters
  • start_date (str) – The start date in format mm/yyyy.

  • end_date (str) – The start date in format mm/yyyy.

Note

The name or the path of the save folder are hard coded and can’t be changed due to some nonsense software engineer decision.

pull_json(JSON_URL)[source]

Download the DODF JSON file available on the current day.

The file is saved either in the path provided or in the default ‘dodf’ directory.

Note

There is no way of downloading JSON files from past days because they are not provided.

Downloader Private Methods

One does not access directly none of those methods, but they are listed here in case the programmer using the downloader library needs more informations.

Path Handling

Methods that handle the creation of the paths to the dowloaded DODFS.

Downloader._create_single_folder(path)[source]

Create a single folder given the directory path.

This function might create a folder, observe that the folder already exists, or raise an error if the folder cannot be created.

Parameters

path (str) – The path to be created

Raises

OSError – Error creating the directory.

Downloader._create_download_folder()[source]

Create Downloaded DODFs Structures.

Downloader._make_month_path(year, actual_date)[source]

Create and return the folder for the year and month being download.

Parameters
  • year (int) – The year respective to the folder.

  • actual_date (datetime) – The date in which the downloaded

  • corresponds. (DODF) –

Returns

The path to the actual month in which the download is being made.

URL Making

Methods that construct an URL to further make the download request. .. .. automethod:: dodfminer.downloader.core.Downloader._make_url .. .. automethod:: dodfminer.downloader.core.Downloader._make_href_url .. .. automethod:: dodfminer.downloader.core.Downloader._make_download_url

Web Requests

Methods that handle the download request and its execution.

Downloader._fail_request_message(url, error)[source]

Log error messages in download.

Parameters
  • url (str) – The failing url to the website.

  • error (str) – The kind of error happening.

Downloader._download_pdf(url, path)[source]

Download the DODF PDF.

Note

Might be time consuming depending on bandwidth.

Parameters
  • url (str) – The pdf url.

  • path (str) – The path to save the pdf.

Raises

RequestException – Error in case the request to download fails.

Others

Other methods for the downloader library.

classmethod Downloader._string_to_date(date)[source]

Convert the date to datetime.

Parameters

date (datetime) – The date to be converted in string format.

Returns

Return the date formated in string now as datetime datatype.

Raises

Exception – date passed through cli is in wrong format.

Downloader._file_exist(path)[source]

Check if a file exists.

Prevents redownloads.

Parameters

path (str) – The path where the file might be

Returns

Boolean indicating if file does really exists.

Downloader._log(message)[source]

Logs a message following the downloader pattern.

Parameters

message (str) – The message to be logged.