Evaluate Trustworthy AIKR objects implemented by machine learning powered servicesThis plan defines the role of the AI KR Strategist.Artificial Intelligence Knowledge Representation Community GroupAIKR CGOrganization_cd4a9bd6-0ec8-425c-ae47-9599f9b4b209Carl MattocksCoChairFor all AI systems to have clearly and transparently documented goals and performance data showing that they are being achieved.Vision_861566c8-e9be-4642-b52f-f673fa499f4eThe mission of an AI Strategist is to define the purpose and goals of AI systems, as well as the KPIs by which we can determine if the system is meeting its goals.Mission_861566c8-e9be-4642-b52f-f673fa499f4eEthicalEnsure AI Systems adhere to pivotal principles, such as, confidentiality, autonomy, accountability and veracityGoal_bbcb3dc4-5946-4d7d-b43f-0a55af305cc21_2e3e2dbe-7e6f-11ea-bb2c-d85ab95fb34a_2e3e3070-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_1[To be determined]Machine Learning EvaluationEvaluate machine learning modelsGoal_56cd3982-542c-4719-965e-0bcce6606a012Artificial Intelligence Knowledge Representation Community Group (AIKR CG)Community of InterestTrustworthyProvide the foundation for a trustworthy AIKRObjective_fa222026-9d57-4423-9433-9933bfe755e01Evaluation metrics are tied to machine learning tasks. Perhaps the easiest metric to interpret is the percent of estimates that differ from the true value by no more than X%._2e3e3232-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_2[To be determined]TrackTrack Classification Performance IndicatorsObjective_964efa5e-58a7-4d9a-a839-daa8aef2a8572Ontological Statement: Classification Accuracy is the ratio of number of correct class label predictions to the total number of input samples data. Ontological Statement: F1 Score measure the Harmonic Mean between precision and recall. The range for F1 Score is [0, 1]. It tells you how precise your classifier is (how many instances it classifies correctly), as well as how robust it is (it does not miss a significant number of instances).6AUC-ROC CurvePerformanceIndicator_d784403b-241c-418c-bd14-7930f884a440PLACEHOLDER_3[To be determined]Ontological Statement: check performance of multi - class classification AUROC (Area Under the Receiver Operating Characteristics) curve.Ontological Statement: Area Under Curve(AUC) is one of the most widely used metrics for evaluation. It is used for binary classification problem. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than a randomly chosen negative example. True Positive Rate (Sensitivity) : True Positive Rate is defined as TP/ (FN+TP). True Positive Rate corresponds to the proportion of positive data points that are correctly considered as positive, with respect to all positive data points. False Positive Rate (Specificity) : False Positive Rate is defined as FP / (FP+TN). False Positive Rate corresponds to the proportion of negative data points that are mistakenly considered as positive, with respect to all negative data points.5Log-LossPerformanceIndicator_102e78ab-4e9a-4d04-8476-06b7121b3294PLACEHOLDER_4[To be determined]Ontological Statement: Logarithmic loss (related to cross-entropy) measures the performance of a classification model where the prediction input is a probability value between 0 and 1 - Log loss increases as the predicted probability diverges from the actual label Logarithmic Loss or Log Loss, works by penalising the false classifications. It works well for multi-class classification. When working with Log Loss, the classifier must assign probability to each class for all the samples. where, y_ij, indicates whether sample i belongs to class j or not p_ij, indicates the probability of sample i belonging to class j Log Loss has no upper bound and it exists on the range [0, ∞). Log Loss nearer to 0 indicates higher accuracy, whereas if the Log Loss is away from 0 then it indicates lower accuracy. In general, minimising Log Loss gives greater accuracy for the classifier.2AccuracyPerformanceIndicator_1611aab4-de88-4a4f-ad30-f74165037856PLACEHOLDER_5[To be determined]Ontological Statement: Classification Rate or Accuracy is given by the relation: True Positives + True Negatives / All Instances (True & False Positives + True & False Negatives)4Per-class accuracyPerformanceIndicator_6d27fb46-89e0-40ca-9fd6-680f760608bdPLACEHOLDER_6[To be determined]3Confusion MatrixPerformanceIndicator_4a78f4f9-6bd5-4382-85c4-d0bfb0c16549PLACEHOLDER_7[To be determined]Ontological Statement: A confusion matrix is a summary of prediction results on a classification problem. The number of correct and incorrect predictions are summarized with count values and broken down by each class (the types of errors being made) Types : * True Positives : The cases in which we predicted YES and the actual output was also YES. * True Negatives : The cases in which we predicted NO and the actual output was NO. * False Positives : The cases in which we predicted YES and the actual output was NO. * False Negatives : The cases in which we predicted NO and the actual output was YES. Accuracy for the matrix can be calculated by taking average of the values lying across the “main diagonal” Type StartDate EndDate Description Target Number of True Positives Target Number of False Positives Target Number of True Negatives Target Number of False Negatives Actual [To be determined]11"Almost correct" predictionsPerformanceIndicator_0ef5a0b6-499e-4128-a3fa-b112e098a49bPLACEHOLDER_8[To be determined]1Precision RecallPerformanceIndicator_25badc58-238a-4cb5-ad8e-c218b425b3a0PLACEHOLDER_9[To be determined]Ontological Statement: Precision is the number of correct positive results divided by the number of positive results predicted by the classifier. Ontological Statement: Recall is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive).9Regression AnalysisPerformanceIndicator_8d8ced68-00f3-4604-a350-bab9e4984375PLACEHOLDER_10[To be determined]Root Mean Square Error (RMSE) Ontological Statement: Root Mean Square Error (RMSE) is the standard deviation of the residuals (prediction errors). Residuals are a measure of how far from the regression line data points are; RMSE is a measure of how spread out these residuals are.8NDCGPerformanceIndicator_2f9b2c7a-892f-4433-8705-00267505f2bcPLACEHOLDER_11[To be determined]Ontological Statement: Normalized discounted cumulative gain (DCG) is a measure of ranking quality. In information retrieval, DCG measures the usefulness, or gain, of a document based on its position in the result list.10Quantiles of ErrorsPerformanceIndicator_5e57c985-7a58-4cf6-b711-2cf7ad3ddd9ePLACEHOLDER_12[To be determined]Quantiles (or percentiles), which is the element of a set that is larger than half of the set, and smaller than the other half.7F-measurePerformanceIndicator_1621ab3f-2e83-484b-95cb-89b63ecb46d9PLACEHOLDER_13[To be determined]F1 Score is the Harmonic Mean between precision and recall. Ontological Statement: F-measure represents both Precision and Recall it helps to have a measurement that represents both of them. F-measure is calculated using Harmonic Mean (in place of Arithmetic Mean). Ontological Statement:  Mean Absolute Error is the average of the difference between the Original Values and the Predicted Values. It gives us the measure of how far the predictions were from the actual output. Ontological Statement:  Mean Squared Error(MSE) takes the average of the square of the difference between the original values and the predicted values.LawfulEnsure AI Systems comply with all applicable laws and regulations, such as, provision audit data defined by a governance operating modelGoal_b71896a0-3d86-4713-a720-15738315e36b3_2e3e32fa-7e6f-11ea-bb2c-d85ab95fb34a_2e3e343a-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_14[To be determined]Ontological StatementsEmploy ontological statements when explaining AIKR object audit data, veracity facts and (human, social and technology) risk mitigation factorsGoal_0083c58a-3d13-4e0e-95d1-8391c3f6414a4_2e3e3598-7e6f-11ea-bb2c-d85ab95fb34a_2e3e36d8-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_15[To be determined]TrackTrack AIKR object performance outcome via KPI (Key Performance Indicator) based on supervised learning models measurementsGoal_e2b04ebe-49d3-43f3-a723-a44135690f645_2e3e3b24-7e6f-11ea-bb2c-d85ab95fb34a_2e3e3cd2-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_16[To be determined]DocumentDocument the vision, values, goals, objectives for one or more AIKR objectsGoal_995c0b60-d64c-445e-86c8-a6f755f5ed9a6_2e3e3e30-7e6f-11ea-bb2c-d85ab95fb34a_2e3e3f8e-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_17[To be determined]RobustEnsure AI Systems are designed to handle uncertainty and tolerate perturbation from a likely threat perspective, such as, design considerations incorporate human, social and technology risk factorsGoal_5a34fa22-8d74-402f-b111-d0e585de11a27_2e3e4146-7e6f-11ea-bb2c-d85ab95fb34a_2e3e42b8-7e6f-11ea-bb2c-d85ab95fb34aPLACEHOLDER_18[To be determined]StrategyPlan_861566c8-e9be-4642-b52f-f673fa499f4e2020-04-012020-04-14https://www.stratnavapp.com/StratML/Part2/861566c8-e9be-4642-b52f-f673fa499f4eSubmitter_861566c8-e9be-4642-b52f-f673fa499f4eCarlMattocksCarlMattocks@WellnessIntelligence.Institute_2e3e4434-7e6f-11ea-bb2c-d85ab95fb34a