Cloud Computing Service Metrics Description - DRAFTThis document proposes a framework that identifies and characterizes the information and relationships needed to describe and observe properties of cloud services that are representative, accurate and reproducible. This information can be used in a variety of ways including, collection, comparison, gap analysis, and assessment or description of metrics at the technical or business levels. These metrics can connect information intended for decision-making, for the service agreements between provider and customer, for the runtime performance measurement and the underlying properties within the providers system. This document may be used as a source of information to better understand metrology within the context of cloud services, and as a framework to describe, collect and access information related to metrics. The measurement process and methodology necessary for performing the measurement of a given cloud property is not the focus of this document.DISCLAIMER - This document has been prepared by the National Institute of Standards and Technology (NIST) and describes technical research in support of the NIST Cloud Computing Program. Certain commercial entities, equipment, or material may be identified in this document in order to describe a concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that these entities, materials, or equipment are necessarily the best available for the purpose.NIST Cloud Computing Reference Architecture and Taxonomy Working GroupCCRATWG_2ec39258-8800-11e4-97b3-73f808c29cb5Information Technology Laboratory The Information Technology Laboratory (ITL) at the National Institute of Standards and Technology (NIST) promotes the U.S. economy and public welfare by providing technical leadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include the development of management, administrative, technical, and physical standards and guidelines for the cost-effective security and privacy of other than national security-related information in Federal information systems. This Special Publication 500-series reports on ITL’s research, guidance, and outreach efforts in Information Technology and its collaborative activities with
industry, government, and academic organizations.NIST Cloud Computing ProgramInformation Technology LaboratoryU.S. Department of CommercePenny Pritzker, SecretaryNational Institute of Standards and TechnologyWillie May, Acting Under Secretary of Commerce for Standards and Technology and Acting DirectorU.S. Government AgenciesThe targeted audience of this document includes but is not limited to: * U.S. Government agencies * Cloud service customers * Cloud service auditors * Cloud service providersCloud Service CustomersCloud Service AuditorsCloud Service ProvidersFrederic de VaulxNIST co-editor --
This document reflects the contributions and discussions by the members of the NIST Cloud Computing Reference Architecture and Taxonomy Working Group - Cloud Service Metrics Sub Group, lead by Frederic de Vaulx of Prometheus Computing, LLC under contract with the U.S. Department of Commerce, National Institute of Standards and Technology, Information Technology Laboratory.
NIST would like to acknowledge the specific contributions from the following contributors:Jane Siegel(CMU)Jacques Durand(Fujitsu)John Calhoon(Microsoft)Jeff Perdue(CMU)Steve Woodward(Cloud Perspectives)Alan Sill(TTU)Ken E. Stavinoha(Solutions Architect, Cisco Systems)Tom Rutt(Fujitsu)Jenny Huang(AT&T)Eric Roggenstroh(eGT for GSA)Omar Fink(SAIC)Steven J. McGee(SAW Concepts LLC)Eric Simmon(NIST)Scott Feuless(ISG/CSMIC)Jim Watts(TSA)Keyun Ruan(Chief Scientist, Espion Group, Ireland)Jesus Luna(CSA)William M. Fitzgerald(EMC Information Systems International, Ireland)Kimberley Laris(Positive Assurance)Massimiliano Rak(CerICT, Italy)Neeraj Suri(Technical University of Darmstadt, Germany)David Núñez(Universidad de Málaga, Spain)Carmen Fernández-Gago(Universidad de Málaga, Spain)Isaac Agudo(Universidad de Málaga, Spain)Eric SimmonNIST co-editorRobert BohnNIST co-editor_2ec39474-8800-11e4-97b3-73f808c29cb5To propose a framework that identifies and characterizes the information and relationships needed to describe and observe properties of cloud services that are representative, accurate and reproducible._2ec395d2-8800-11e4-97b3-73f808c29cb5ClarityWith cloud computing in the mainstream, there is a preponderance of cloud based services in the market and the choices for consumers increase daily. However, comparing the service offerings between cloud service providers is not a straightforward exercise. To be successful in procuring cloud services, one must have requirements that are clear, create service level agreements (SLA) which reflect these requirements and be measureable in order to validate the delivery of these services along with their performance and remedies.QualityAs part of the decision making framework for moving to the cloud, having data on measurable capabilities, for example - quality of service, availability and reliability, give the cloud service customer the tools and opportunity to make informed choices and to gain an understanding of the service being delivered. NIST’s definition of cloud computing describes a “Measured Service” as being one of the five essential characteristics of the cloud computing model. To describe a “measured service”, one needs to identify the cloud service properties that have to be measured and what their standards of measurement or metrics are.AvailabilityReliabilityMetricsA metric provides knowledge about characteristics of a cloud property through both its definition (e.g. expression, unit, rules) and the values resulting from the observation of the property. For instance, a customer response time metric can be used to estimate a specific response time property (i.e. response time from customer to customer) of a cloud email service search feature. It also provides the necessary information that is needed for to reproduce and verify observations and measurement results. In this context, the role of that metrics play is very important to support decision-making as well as: * Selecting cloud services * Defining and enforcing service agreements * Monitoring cloud services * Accounting and AuditingScenariosA scenario represents a particular use case in which metrics play a role. Stakeholders need to have a way to understand, assess, compare, combine and make decisions about cloud service properties. This means that for a given scenario (e.g. choosing a cloud service or setup a service agreement), a stakeholder needs to be able to get information on cloud service properties, which when measured (observed) will guide the stakeholder along the proper course of action. The scenario and cloud service property will determine the metric (standard of measurement) to be used.SelectionSelect a service_2ec39690-8800-11e4-97b3-73f808c29cb51Cloud Services Measurement Initiative ConsortiumMetrics for Selecting Cloud Services -- Metrics are essential at the stage of deciding what cloud offering should be best suited to meet the business and technical requirements. The customer of cloud services should be able to select and use metrics and their underlying measures to assess and decide which offering would be best. Solutions like the Service Measurement Index (SMI), produced by the Cloud Services Measurement Initiative Consortium (CSMIC), could be used to determine which metrics are relevant to the selection of a particular cloud offering._2ec3974e-8800-11e4-97b3-73f808c29cb5AgreementAgree upon service levels_2ec39884-8800-11e4-97b3-73f808c29cb52Metrics for Service Agreements (SAs) -- A Service Agreement (SA) represents a binding agreement between the provider and customer of a cloud service. Among the elements that it contains are the description of the service, the rights and responsibilities of both the provider and the customer and terms definition. It also contains essential information related to the measurement of different aspects of the cloud service (e.g., its business level objectives or its performance level). The definition and usage of appropriate metrics with their underlying measures are essential components of the Service Level Agreement (SLA) and Service Level Objectives (SLO), which are constituents of the SA. The references [9] and [10] describe, in detail, the importance of and need for metrics in SLAs. At this point, the
metrics are used to set the boundaries and margins of errors the provider of the service abides by and sets their limitations. For instance these metrics could be used at runtime for service monitoring and balancing, or remediation (e.g., financial). Using a standardized set of metrics or metric templates in SAs makes it easier and quicker to define SLAs and SLOs, and to compare them with others._2ec3996a-8800-11e4-97b3-73f808c29cb5VerificationVerify service levels_2ec39a28-8800-11e4-97b3-73f808c29cb53Metrics for Service Measurement -- Once the customer purchases a cloud service, it is necessary to ensure the service level objectives are being met. If they are not met, a pre-determined remedy needs to be initiated._2ec39f50-8800-11e4-97b3-73f808c29cb52014-12-19OwenAmburOwen.Ambur@verizon.net