tasty

console

tasty.console.generate_input(args)

Generate a CSV input file with shape names as headers. Doesn’t use the ttl shacl files, instead uses the:

• source_shapes/*.json

Parameters

args –

Returns

tasty.console.generate_shapes(args)

Generate a SHACL shapes file for each JSON source file. This is done on a schema basis, i.e. all haystack or brick source shapes are sourced from tasty/source_shapes/<schema-name>/*.json :param args: :return:

tasty.console.main()

Main launch point for the CLI. Mainly points to other functions.

tasty.console.validate(args)

Validate an input data graph using the marked up csv file. The csv file should have X’s in entity rows to indicate it should be validated against a specific shape of interest. :param args: :return:

entities

class tasty.entities.BrickEquipmentDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Brick equipment types. Attributes are only added upon calling the ‘bind’ method.

class tasty.entities.BrickLocationDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Brick equipment types. Attributes are only added upon calling the ‘bind’ method.

class tasty.entities.BrickPointDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Brick point types. Attributes are only added upon calling the ‘bind’ method.

class tasty.entities.BrickRefDefs(version, include_inverse=True)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to Haystack object properties Attributes are only added upon calling the ‘bind’ method.

bind() → None

Create an attribute for each first class type. The value of each attribute is an EntityType. :return:

class tasty.entities.BrickSystemDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Brick equipment types. Attributes are only added upon calling the ‘bind’ method.

class tasty.entities.BrickZoneDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Brick equipment types. Attributes are only added upon calling the ‘bind’ method.

class tasty.entities.CompositeShape(data, schema, version)

Bases: object

apply_shape_mixins(equip_id, namespace, ref, entity, optional_points=False)

cast_to_entity(id=None) → tasty.entities.EntityType

class tasty.entities.EntityDefs(schema: str, version: str)

Bases: object

A base class giving access to first class ontological types via simple attributes.

bind() → None

Create an attribute for each first class type. The value of each attribute is an EntityType. :return:

find(to_find: Union[str, list], case_sensitive=False) → List

Given a string or list of strings, return attributes of the class (corresponding to first class entities) where the string(s) are present. :param to_find: :param case_sensitive: match case in the search :return: a list of attribute names matching the search

class tasty.entities.EntityType(type_uri: rdflib.term.URIRef, type_docs: rdflib.term.Literal, schema, version, namespace: rdflib.namespace.Namespace = None)

Bases: object

A generic entity type, identified via its type_uri. This should be considered as an instance of a first class type from one of the ontologies, i.e. a point, ahu, etc.

add_relationship(predicate: tasty.entities.RefType, obj: tasty.entities.EntityType)

add_tags(tags: List[str], ontology: rdflib.graph.Graph)

bind_to_graph(g: rdflib.graph.Graph) → bool

Adds a triple representing itself to the graph. :param g: the graph to add the triple to :return: a bool indicating success

deep_copy() → tasty.entities.EntityType

Return a deep copy of the current self

set_id(new_id=None) → uuid.UUID

Set the id or generate a :return: the uuid

set_namespace(ns: Union[str, rdflib.namespace.Namespace]) → bool

Set the _namespace for this specific entity. :param ns: :return: a bool indicating success

set_node_name()

Sets the node name based on the _id and _namespace if not yet set

sync(touched_nodes=[])

type_docs() → str

type_uri() → str

class tasty.entities.HaystackEquipDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Haystack equipment types Attributes are only added upon calling the ‘bind’ method.

bind() → None

Create an attribute for each first class type. The value of each attribute is an EntityType. :return:

class tasty.entities.HaystackPointDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to first class Haystack point types Attributes are only added upon calling the ‘bind’ method.

bind() → None

Create an attribute for each first class type. The value of each attribute is an EntityType. :return:

class tasty.entities.HaystackRefDefs(version)

Bases: tasty.entities.EntityDefs

A class with attributes corresponding to Haystack object properties Attributes are only added upon calling the ‘bind’ method.

bind() → None

Create an attribute for each first class type. The value of each attribute is an EntityType. :return:

class tasty.entities.RefType(type_uri: rdflib.term.URIRef, type_docs: rdflib.term.Literal)

Bases: object

class tasty.entities.ShapesWrapper(schema, version)

Bases: object

add_shapes(shapes)

bind()

bind_composite()

evaluate_shape(shape)

class tasty.entities.SimpleShape(data, schema, version)

Bases: object

apply(equip_id, namespace, parent, equip_ref)

cast_to_entity(id=None) → tasty.entities.EntityType

exceptions

exception tasty.exceptions.MultipleTermsFoundError(message)

Bases: tasty.exceptions.TastyError

exception tasty.exceptions.TastyError(message)

Bases: Exception

exception tasty.exceptions.TemplateRegistrationError(message)

Bases: tasty.exceptions.TastyError

exception tasty.exceptions.TemplateValidationError(message)

Bases: tasty.exceptions.TastyError

exception tasty.exceptions.TermNotFoundError(message)

Bases: tasty.exceptions.TastyError

generate_input_file

tasty.generate_input_file.generate_input_file(shape_files: List, data_graph: str, output_file: str, composite: bool = True)

Generate a csv file for users to input data. The shape name column headers are added based on the shape files provided. entity ids and names are populated from the data file provided.

entity-id | entity-name | shape-name-1 | …

…id1… | ..a name.. | | ..

Parameters

• shape_files – paths to JSON source_shapes files to use to populate the csv headers

• data_graph – name of data graph to read in to populate entity-id and entity-name columns

• output_file – path to output file to write

• composite – whether to only consider more complex shapes

Returns

graphs

tasty.graphs.bind_prefixes(graph: rdflib.graph.Graph) → None

Associate common prefixes with the graph

tasty.graphs.bind_versioned_prefixes(graph: rdflib.graph.Graph, schema: str, version: str) → None

Parameters

• graph – [rdflib.Graph]

• schema – [str] A valid key from SUPPORTED_SCHEMAS

• version – [str] A valid version from SUPPORTED_SCHEMAS

Returns

tasty.graphs.get_namespaced_term(ontology: rdflib.graph.Graph, term: str) → Union[rdflib.term.URIRef, bool]

Return a fully namespaced term if it exists exclusively in the ontology, else return False :param ontology: :param term: :return:

tasty.graphs.get_namespaces_given_term(ontology: rdflib.graph.Graph, term: str) → List[rdflib.namespace.Namespace]

Return a list of Namespaces where this term exists in the provided Graph. The hope is for the len to be 1, i.e. there is just one Namespace where this term exists :param ontology: [Graph] an ontology (Brick or Haystack) pre :param term: [str] a term to search for in the Namespace :return: [list[Namespace]]

tasty.graphs.get_versioned_graph(schema: str, version: str) → rdflib.graph.Graph

Get an empty Graph with the correct namespaces loaded

Parameters

• schema – [str] A valid key from SUPPORTED_SCHEMAS

• version – [str] A valid version from SUPPORTED_SCHEMAS

Returns

tasty.graphs.graph_to_hayson_string(graph: rdflib.graph.Graph) → str

Return the Haystack JSON (Hayson) encoding of an RDF graph. :param graph: [rdflib.Graph] :return: [str]

tasty.graphs.has_one_namespace(ns)

Run after tg.get_namespaces_given_term to validate only a single ns was found :param ns: [List[Namespace]] :param candidate: [str] :return:

tasty.graphs.is_valid_schema_and_version(schema: str, version: str) → bool

Check that schema and version are supported. Raise exceptions if not. :param schema: [str] A valid key from SUPPORTED_SCHEMAS :param version: [str] A valid version from SUPPORTED_SCHEMAS :return: [bool]

tasty.graphs.load_ontology(schema: str, version: str) → rdflib.graph.Graph

Load an ontology and return as Graph with the correct namespaces :param schema: [str] A valid key from SUPPORTED_SCHEMAS :param version: [str] A valid version from SUPPORTED_SCHEMAS :return:

shapes_generator

class tasty.shapes_generator.ShapesGenerator(schema, version)

Bases: object

add_all_mixins(namespaced_shape: rdflib.term.URIRef, shape: dict)

Add all mixins to the provided namespaced_shape. :param namespaced_shape: :param shape: :param ns: namespace to use for the mixin :return:

add_all_tags(shape_map: Dict, namespaced_shape: rdflib.term.URIRef, context: str) → int

Considers both ‘tags’ and ‘tags-custom’ keys. :param context: :param shape_map: :param namespaced_shape: :return:

add_all_types(shape_map: Dict, namespaced_shape: rdflib.term.URIRef) → int

Loop through the ‘types’ in the shape_map and add a sh:class constraint for each. The types are not expected to be namespaced, but should exist in the ontology. :param shape_map: :param namespaced_shape: :return:

add_min_count_for_required_nodes(each_path: dict, namespaced_shape: rdflib.term.URIRef, namespaced_path: rdflib.term.URIRef)

Count the required number of ‘types’ and ‘shapes’ and add a minCount property constraint. Considers

inverse paths as well. Something like:

sh:property [ sh:path [ sh:inversePath phIoT:equipRef ] ; sh:minCount 3 ;

] ;

Parameters

• each_path –

• namespaced_shape –

• namespaced_path –

Returns

add_predicates(namespaced_shape, predicates: list, required=True)

add_sh_path(property_bn: rdflib.term.BNode, each_path: dict, namespaced_path)

Add a path constraint, either inverse or direct. Looks like:

[ sh:path [ sh:inversePath phIoT:equipRef ] ] (inverse) [ sh:path phIoT:equipRef ] (direct) where the outer is the property_bn

Parameters

• property_bn – blank node of the current property

• each_path –

• namespaced_path – the predicate to be traversed, i.e. ph:hasTag, phIoT:equipRef

Returns

add_shapes_and_types(parent_namespaced_shape: rdflib.term.URIRef, namespaced_path_from_parent_to_children: rdflib.term.URIRef, each_path: dict, required=True)

For a given path provided, transform all of the ‘shapes’ and ‘types’ from the source shape definition into SHACL. Handles optional by setting the sh:severity to sh:Warning :param parent_namespaced_shape: :param namespaced_path: example: URIRef(‘https://project-haystack.org/def/phIoT/3.9.10#equipRef’) :param each_path: :param required: :return:

add_tags_types_and_predicates(ns_shape: rdflib.term.URIRef, shape: dict)

Take the provided source shape dict object and transform to a SHACL shape: - tag requirements will be added as property shape requirements to the shape itself with a ph:hasTag traversal - type requirements will be added as sh:class constrants to the shape itself - predicates will be added as property shape requirements to the shape itself with a traversal defined by

the ‘path’ and ‘path-type’ keys.

Parameters

• ns_shape –

• shape –

Returns

get_namespaced_shape(shape_name)

load_all_source_shapes_by_schema()

Called in initialization. Based on <schema> provided, looks in the source_shapes/<schema> dir for all JSON files with that schema name. - Loads into self.source_shapes_by_file - creates self.shape_lookup:

sg.shapes_lookup.get(‘damper-cmd-shape’) # {‘namespace’: ‘https://project-haystack.org/datashapes/core#’, ‘prefix’: ‘phShapes’} sg.shapes_lookup.get(‘foo’) # NoneType

Returns

main(source_shape_full: dict)

Given a full source shape as a dict, generate SHACL shapes for all items under ‘shapes’ list. :param source_shape_full: :param g: :param ontology: :return:

main_generate_all_and_merge()

reset_shapes_graph()

stub_new_qualified_value_property(each_path: dict, parent_namespaced_shape: rdflib.term.URIRef, namespaced_path: rdflib.term.URIRef) → rdflib.term.BNode

Create a new property as a BNode, where the property specifies qualified constraints.

Output looks like:

[ sh:path [ sh:inversePath phIoT:equipRef ] ;

sh:qualifiedMaxCount 1 ; sh:qualifiedMinCount 1 ; sh:qualifiedValueShapesDisjoint true ] ;

The sh:qualifiedValueShape added elsewhere. Assumes we want:

• exactly 1 of the shapes

• distinctness (i.e. disjoint)

Parameters

• each_path –

• parent_namespaced_shape –

• namespaced_path –

Returns

write_shapes_graph_to_generated_shapes_dir(file_name: str)

Serialize the provided shapes graph to the tasty/generated_shapes/ directory. :param file_name: name of the output file to write :return:

shapes_loader

class tasty.shapes_loader.ShapesLoader(schema=None)

Bases: object

Wrapper class to merge all SHACL shape files into a single RDF Graph. Can optionally merge only files specific to a schema, i.e. Haystack or Brick.

load_all_shapes() → rdflib.graph.Graph

templates

class tasty.templates.BaseTemplate(**kwargs)

Bases: object

static has_minimum_keys(template)

populate_template_basics() → None

Only populate template if valid :return:

validate_template_against_schema(schema_path='/home/docs/checkouts/readthedocs.org/user_builds/tasty/envs/latest/lib/python3.7/site-packages/tasty/schemas/template.schema.json', template_type=None) → None

Validate self._template (dict) against the JSON schema. :param schema_path: [str] full/path/to/schema.json :return:

class tasty.templates.EntityTemplate(entity_classes: Set[Tuple[rdflib.namespace.Namespace, str]], schema_name: str, schema_version: str, typing_properties: Set, properties: Set[Tuple[rdflib.namespace.Namespace, str, dict]])

Bases: object

classmethod find_with_class(namespaced_class: Tuple[rdflib.namespace.Namespace, str])

Search all registered templates and return the objects with the class defined. :param namespaced_class: [Tuple[Namespace, str]] A 2-term tuple, where the str term represents the

class to find, something like ‘cur-point’ or ‘Discharge_Air_Flow_Sensor’

Returns

[List[EntityTemplate]] a list of EntityTemplate objects matching the description

classmethod find_with_classes(namespaced_classes: Set[Tuple[rdflib.namespace.Namespace, str]])

Search all registered templates and return the objects with atleast the classes defined.

Example: namespaced_classes = {(NS1, cur-point), (NS1, his-point)} entity_classes1 = {(NS1, cur-point), (NS1, his-point), (NS1, writable-point)} -> would be added entity_classes2 = {(NS1, cur-point)} -> would not be added

Parameters

namespaced_classes – [Set[Tuple[Namespace, str]]] A set of 2-term tuples, where the str term represents the class to find, something like ‘cur-point’ or ‘Discharge_Air_Flow_Sensor’

Returns

[List[EntityTemplate]] a list of EntityTemplate objects matching the description

classmethod get_equivalent(entity_classes: Set, schema_name: str, schema_version: str, typing_properties: Set, properties: Set[Tuple[rdflib.namespace.Namespace, str, dict]])

get_namespaces() → Set[rdflib.namespace.Namespace]

Get all unique Namespaces for the entity template :return: [Set[Namespace]]

get_simple_classes() → Set[str]

Just get terms, no namespaces :return: [Set[str]]

get_simple_properties() → dict

Get other_properties as hash map of keys, values, no namespaces :return: [dict]

get_simple_typing_info() → Set[str]

Get terms for the entity_class and typing_properties, no namespaces :return: [Set[str]]

instances = {}

classmethod register_template(template)

Add the template to the set of templates available :param template: [EntityTemplate] the EntityTemplate to register :return:

validate_data()

class tasty.templates.EquipmentTemplate(**kwargs)

Bases: tasty.templates.BaseTemplate

Template for a piece of equipment based on a class definition from Project Haystack or Brick Schema. The ‘base class’ is defined by the ‘extends’ key in the template. This must resolve to: - Haystack: rdfs:subClassOf* phIoT:equip - Brick: rdfs:subClassOf* brick:Equipment The Equipment template defines expected telemetry point types to associate with it, either through:

• a PointGroupTemplate symbol (i.e. SD).

• a telemetry_point_type definition (i.e. SD).

get_all_points_as_entity_templates() → Set[tasty.templates.EntityTemplate]

instances = {}

classmethod register_template(template)

Register the template to make it available externally. :param template: [EquipmentTemplate] :return:

resolve_extends() → None

Resolve the value of ‘extends’ to a valid equipment class. Sets self.extends = (Namespace, term) when found. :return:

resolve_telemetry_point_types()

class tasty.templates.PointGroupTemplate(**kwargs)

Bases: tasty.templates.BaseTemplate

add_telemetry_point_to_template(entity_template: tasty.templates.EntityTemplate) → None

Update the set of telemetry_point_entity_templates with a new EntityTemplate :param entity_template: [EntityTemplate] a new entity to add to this point group :return:

classmethod find_given_symbol(symbol)

Find all PointGroupTemplates with the given symbol. This will likely return multiple, as symbols themselves may only be unique in the context of a given [symbol, schema, version] set :param symbol: [str] :return: [List[PointGroupTemplate]]

classmethod find_given_symbol_schema_version(symbol, schema_name, schema_version)

Find all PointGroupTemplates with the given characteristics. This should hopefully resolve to just 1, although not guaranteed. TODO: decide if a unique combination of [symbol, schema_version, schema_name] should

occur only once. Initial thought is YES.

Parameters

• symbol – [str]

• schema_name – [str]

• schema_version – [str]

Returns

[List[PointGroupTemplate]]

instances = {}

populate_template_basics() → None

See BaseTemplate.populate_template_basics Registers the Class if valid. :return:

classmethod register_template(template) → None

Register the template to make it available externally. :param template: [PointGroupTemplate] :return:

resolve_telemetry_point_types() → None

Wrapper around: resolve_telemetry_points_to_entity_templates. Uses keys found in the template to run. :return:

write(file_path: str) → None

Write the _template data to the file as yaml. :param file_path: [str] full/path/to/file.yaml :return:

tasty.templates.get_namespaced_terms(ontology: rdflib.graph.Graph, terms: [<class 'str'>, <class 'dict'>]) → Set

TODO: document function :param ontology: [Graph] A loaded ontology :param terms: [str] A Brick class, such as ‘Discharge_Air_Temperature_Sensor’ a set of Haystack tags,

such as ‘discharge-air-temp-sensor-point’, or a dict of properties, such as: {field1: {_kind: number}, field2: {val: cfm}}

Returns

tasty.templates.get_prefix(namespaces, term)

Return the prefix for the term given namespaces in the ontology. :param namespaces: [List[Namespace]] :param term: [str] term :return: [str] if found, the prefix :return: [None] if not found

tasty.templates.hget_entity_classes(ontology, candidates)

Given a ‘string-of-haystack-tags’, determine valid classes, markers, and properties. See return. :param ontology: [Graph] a loaded ontology :param candidates: [str] a ‘-’ delimited string, where each term is a haystack concept to figure out :return: [dict[str, Set]] The returned dict has structure as follows:

{

A set of namespaced valid entity classes ‘classes’: {(Namespace, term), (Namespace, term) …},

A set of namespaced valid markers NOT used in entity class typing. These must subClass* from ph:marker ‘markers’: {(Namespace, term), …}

A set of namespaced properties. These are terms that are NOT markers, but are Datatype Properties. The third term in the tuple always specifies a ‘val’ of None (i.e. just expect the property to be defined) ‘properties’: {(Namespace, term, frozendict({‘val’: None})), …}

}

tasty.templates.load_template_file(path_to_file: str) → List[dict]

Read in a template file and return templates :param path_to_file: [str] :return:

tasty.templates.load_template_schema(path_to_file: str) → dict

Load in the template schema and return :param path_to_file: [str] :return:

tasty.templates.resolve_telemetry_points_to_entity_templates(telemetry_point_types: dict, schema_name: str, version: str) → Set[tasty.templates.EntityTemplate]

Resolve each telemetry point (a key in the dict) to an EntityTemplate and return the set of created entity templates. Used in the PointGroupTemplate validator :param telemetry_point_types: [dict] :param schema_name: [str] One of the supported Schema names, see tasty/schemas/template.schema.json :param version: :return:

tasty.templates.resolve_to_entity_template(ont, typing_metadata, properties, schema_name, version)

tasty.templates.validate_template_against_schema(instance: dict, schema: dict) → Tuple[bool, str]

Validate a single template against the template schema :param instance: [dict] the template to validate :param schema: [dict] the schema to validate against :return:

validate

tasty.validate.pretty_print_errors(results_graph: rdflib.graph.Graph) → None

Print out errors (sh:Violation) vs. warnings (sh:Warning) given a results graph. Grabs the focus node (i.e. what the error fired on) and the shape that was fired.

Parameters

results_graph – graph used to generate the output

Returns

tasty.validate.validate_from_csv(data_graph: str, input_file: str) → None

Given a csv as generated by generate_input_file, add the marked entities as target nodes for the specific shapes and run through a SHACL validator. Merges all ttl files from tasty/generated_shapes into a single graph for simplicity.

Parameters

• data_graph – path to a data graph to load

• input_file – path to the input csv file

Returns