Note
This is a public alpha release, and therefore features and functionality may change and the software and documentation may contain technical bugs or other issues. If you discover any issues please consider registering a GitHub issue.
datasources.connectors.base module¶
This module contains base classes for data connectors.
Data connectors are the component of PEDASI which interacts directly with data provider APIs.
-
class
datasources.connectors.base.
AuthMethod
[source]¶ Bases:
enum.IntEnum
Authentication method to be used when performing a request to the external API.
-
BASIC
= 1¶
-
HEADER
= 2¶
-
NONE
= -1¶
-
UNKNOWN
= 0¶
-
-
class
datasources.connectors.base.
BaseDataConnector
(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, **kwargs)[source]¶ Bases:
object
Base class of data connectors which provide access to data / metadata via an external API.
DataConnectors may be defined for sources which provide:
- A single dataset
- A data catalogue - a collection of datasets
-
description
= None¶ Help string to be shown when a data provider is choosing a connector
-
static
determine_auth_method
(url: str, api_key: str) → datasources.connectors.base.AuthMethod[source]¶ Determine which authentication method to use to access the data source.
Test each known authentication method in turn until one succeeds.
Parameters: - url – URL to authenticate against
- api_key – API key to use for authentication
Returns: First successful authentication method
-
get_metadata
(params: Union[Mapping[str, str], NoneType] = None)[source]¶ Get metadata from this source using the appropriate API.
Parameters: params – Optional query parameter filters Returns: Requested metadata
-
get_response
(params: Union[Mapping[str, str], NoneType] = None)[source]¶ Transparently return a response from a source API.
Parameters: params – Optional query parameter filters Returns: Requested data / metadata - response is passed transparently
-
is_catalogue
= None¶ Does this data connector represent a data catalogue containing multiple datasets?
-
request_count
¶
-
class
datasources.connectors.base.
DataCatalogueConnector
(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, **kwargs)[source]¶ Bases:
datasources.connectors.base.BaseDataConnector
,collections.abc.Mapping
Base class of data connectors which provide access to a data catalogue.
-
get_datasets
(params: Union[Mapping[str, str], NoneType] = None) → List[str][source]¶ Get the list of datasets provided by this catalogue.
Parameters: params – Query parameters to pass to data source API Returns: List of datasets provided by this catalogue
-
is_catalogue
= True¶ Does this data connector represent a data catalogue containing multiple datasets?
-
-
class
datasources.connectors.base.
DataSetConnector
(location: str, api_key: Union[str, NoneType] = None, auth: Union[Callable, NoneType] = None, metadata: Optional = None)[source]¶ Bases:
datasources.connectors.base.BaseDataConnector
Base class of data connectors which provide access to a single dataset.
Metadata may be passed to the constructor if it has been collected from a previous source otherwise attempting to retrieve metadata will raise NotImplementedError.
If you wish to connect to a source that provides metadata itself, you must create a new connector class which inherits from this one.
-
description
= 'This connector is the default option and should be used when accessing an API at a single endpoint which may or may not accept query parameters.'¶ Help string to be shown when a data provider is choosing a connector
-
get_data
(params: Union[Mapping[str, str], NoneType] = None)[source]¶ Retrieve the data from this source.
If the data is JSON formatted it will be parsed into a dictionary - otherwise it will be passed as plain text.
Parameters: params – Query parameters to be passed through to the data source API Returns: Data source data
-
is_catalogue
= False¶ Does this data connector represent a data catalogue containing multiple datasets?
-
-
exception
datasources.connectors.base.
DatasetNotFoundError
[source]¶ Bases:
Exception
Exception raised when a requested dataset cannot be found within a data source.
-
class
datasources.connectors.base.
HttpHeaderAuth
(username, password)[source]¶ Bases:
requests.auth.HTTPBasicAuth
Requests Auth provider.
The same as HttpBasicAuth - but don’t convert to base64
Used for e.g. Cisco HyperCat API
-
class
datasources.connectors.base.
InternalDataConnector
[source]¶ Bases:
datasources.connectors.base.ReadOnlyInternalDataConnector
Abstract mixin representing a connector for an internally hosted data source which is able to be written to.