Note

This is a public alpha release, and therefore features and functionality may change and the software and documentation may contain technical bugs or other issues. If you discover any issues please consider registering a GitHub issue.

datasources.connectors.base module¶

This module contains base classes for data connectors.

Data connectors are the component of PEDASI which interacts directly with data provider APIs.

class datasources.connectors.base.AuthMethod[source]¶

Bases: enum.IntEnum

Authentication method to be used when performing a request to the external API.

BASIC = 1¶

HEADER = 2¶

NONE = -1¶

UNKNOWN = 0¶

class datasources.connectors.base.BaseDataConnector(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, **kwargs)[source]¶

Bases: object

Base class of data connectors which provide access to data / metadata via an external API.

DataConnectors may be defined for sources which provide:

A single dataset
A data catalogue - a collection of datasets

description = None¶: Help string to be shown when a data provider is choosing a connector

static determine_auth_method(url: str, api_key: str) → datasources.connectors.base.AuthMethod[source]¶

Determine which authentication method to use to access the data source.

Test each known authentication method in turn until one succeeds.

Parameters:	url – URL to authenticate against api_key – API key to use for authentication
Returns:	First successful authentication method

get_metadata(params: Union[Mapping[str, str], NoneType] = None)[source]¶

Get metadata from this source using the appropriate API.

Parameters:	params – Optional query parameter filters
Returns:	Requested metadata

get_response(params: Union[Mapping[str, str], NoneType] = None)[source]¶

Transparently return a response from a source API.

Parameters:	params – Optional query parameter filters
Returns:	Requested data / metadata - response is passed transparently

is_catalogue = None¶: Does this data connector represent a data catalogue containing multiple datasets?

request_count¶

class datasources.connectors.base.DataCatalogueConnector(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, **kwargs)[source]¶

Bases: datasources.connectors.base.BaseDataConnector, collections.abc.Mapping

Base class of data connectors which provide access to a data catalogue.

get_data(params: Union[Mapping[str, str], NoneType] = None)[source]¶

get_datasets(params: Union[Mapping[str, str], NoneType] = None) → List[str][source]¶

Get the list of datasets provided by this catalogue.

Parameters:	params – Query parameters to pass to data source API
Returns:	List of datasets provided by this catalogue

is_catalogue = True¶: Does this data connector represent a data catalogue containing multiple datasets?

class datasources.connectors.base.DataSetConnector(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, metadata: Optional = None)[source]¶

Bases: datasources.connectors.base.BaseDataConnector

Base class of data connectors which provide access to a single dataset.

Metadata may be passed to the constructor if it has been collected from a previous source otherwise attempting to retrieve metadata will raise NotImplementedError.

If you wish to connect to a source that provides metadata itself, you must create a new connector class which inherits from this one.

description = 'This connector is the default option and should be used when accessing an API at a single endpoint which may or may not accept query parameters.'¶: Help string to be shown when a data provider is choosing a connector

get_data(params: Union[Mapping[str, str], NoneType] = None)[source]¶

Retrieve the data from this source.

If the data is JSON formatted it will be parsed into a dictionary - otherwise it will be passed as plain text.

Parameters:	params – Query parameters to be passed through to the data source API
Returns:	Data source data

is_catalogue = False¶: Does this data connector represent a data catalogue containing multiple datasets?

exception datasources.connectors.base.DatasetNotFoundError[source]¶

Bases: Exception

Exception raised when a requested dataset cannot be found within a data source.

class datasources.connectors.base.HttpHeaderAuth(username, password)[source]¶

Bases: requests.auth.HTTPBasicAuth

Requests Auth provider.

The same as HttpBasicAuth - but don’t convert to base64

Used for e.g. Cisco HyperCat API

class datasources.connectors.base.InternalDataConnector[source]¶

Bases: datasources.connectors.base.ReadOnlyInternalDataConnector

Abstract mixin representing a connector for an internally hosted data source which is able to be written to.

clear_data()[source]¶: Clear all data from this data source.

post_data(data: Union[MutableMapping[str, str], List[MutableMapping[str, str]]])[source]¶

Add data to this data source.

Parameters:	data – Data to add

class datasources.connectors.base.ReadOnlyInternalDataConnector[source]¶

Bases: abc.ABC

Abstract mixin representing a connector for an internally hosted data source which is read only.

clean_data(**kwargs)[source]¶: Clean, validate or otherwise structure the data contained within this data source.

class datasources.connectors.base.RequestCounter(count: int = 0)[source]¶

Bases: object

count()[source]¶