Audio version of the article
Introduction
Having “done the business” in clause 8 it is time to check performance. The usual suspects appear here. The organization determines what, how, and when things are to be monitored, measured, analyzed, and evaluated. Add internal audit and management review to the mix and everything expected is addressed. Internal audits provide information on whether the management system conforms to the requirements of the organization and the standard and is effectively implemented and maintained. Management review addresses the question: ‘is the management system suitable, adequate and effective?’ Once again, the auditor should benefit from a consistent set of requirements for checking results against the plan. There is a long list of objective evidence that can be identified and confirmed: metrics, schedules, evaluations, nonconformities and corrective actions, monitoring and measurement results, and audit and management review results.
Clause 9 Performance evaluation has three subclause
- 9.1 Monitoring, measurement, analysis and evaluation
- 9.1.1 General
- 9.1.2 Customer satisfaction
- 9.1.3 Analysis and evaluation
- 9.2 Internal audit
- 9.3 Management Review
- 9.3.1 General
- 9.3.2 Management review inputs
- 9.3.3 Management review outputs
Clause 9.2 Internal audit is a separate article.
9.1 Monitoring, measurement, analysis and evaluation
9.1.1 General
The organization should determine what needs to be monitored and measured. It must also determine the methods for monitoring, measurement, analysis, and evaluation needed to ensure valid results. When the monitoring and measuring must be performed. Also when the results from monitoring and measurement must be analyzed and evaluated. The organization should also evaluate the performance and effectiveness of the quality management system. It must retain appropriate documented information as evidence of the results.
The organization needs to determine what needs to be monitored and measured. The methods were chosen for monitoring, measuring, analysis, and evaluation to ensure valid results; which places a greater emphasis on producing comparable and reproducible results than previously. In addition, the organization should determine when the monitoring and measuring shall be performed, and when the results will be analyzed and evaluated. Note that an organization may have several measurements related to information needs, and these needs may change over time. For example, when a management system is relatively new, it may be important just to monitor the attendance at, say, awareness training. Once the intended rate has been achieved, the organization might look more towards the implementation and quality of the awareness training. It might do this by setting specific awareness objectives and determining the extent to which the attendees have understood what they have learned. Later still, the information need may extend to determine what impact this level of awareness has on performance for the organization (effectiveness). A range of metrics and measurements needs to be developed, collated, and reviewed on a regular basis, and included in the management review process. They need to focus on discipline-specific performance as well as the effectiveness of the management system.
The ISO 9001:2015 goes into more depth and emphasis measuring and evaluating how well an organization’s QMS is performing. A range of metrics and measurements needs to be developed, collated, and reviewed on a regular basis and included in the management review process. In ISO 9001:2008 Clause 8.1 there was a requirement for planning. This has been replaced with the determination of what needs monitoring and measuring, the methods to be used, when performing/analyzing/evaluating. The methods chosen must produce comparable and reproducible results to be considered valid. In addition, the organization should determine when the monitoring and measuring shall be performed, when the results will be analyzed and evaluated.
You must plan and implement processes that monitor, measure, analyze and evaluate the health of your QMS. The focus of these processes must be on product/service conformity, process conformity, and improving QMS effectiveness. Consider using a variety of methods including statistical techniques. In planning what to track and measure, let us review the quality objectives we established and all of the performance indicators we established for each of our QMS processes and activities. You must be careful not to overwhelm your organization with objectives as this may cause more frustration than positive results. Start with objectives that focus on meeting customer requirements and then slowly develop meaningful objectives for key processes and risk-prone processes, as initial targets are achieved. Planning of measurement and data analyses processes must consider the methods and resources such as time, manpower, computer, software, statistical tool, etc needed to collect, organize and analyze product and QMS performance data. Measurement involves physically measuring product characteristics or process parameters against acceptance criteria at predefined intervals and sampling sizes, using predefined measurement devices. Measurement results may not always be fully recorded. Use your organization’s cross-functional knowledge of customer requirements, product, technology, manufacturing processes, etc., to determine what statistical methods to use for each process and to what extent to use them. Include these methods in your quality plan. Statistical methods to verify product characteristics and process parameters include process capability studies, control charts, Pareto analysis, variation analysis. Define and implement appropriate training and competency requirements for all personnel using statistical methods, tools, and analysis. Monitoring usually involves conducting ongoing periodic checks to determine whether product characteristics or process parameters are within acceptable limits. The frequency of monitoring may vary on the risk and reliability of products and processes. Monitoring is also useful in determining the scope and frequency of product and process measurement. The results of monitoring may or may not be recorded. You must identify and document all processes addressing this clause as part of your QMS. For these processes, you must also identify what specific documents, controls, and resources are needed. You could use a product quality plan, documented procedure, or other combination of specific practices, procedures, documents, and methods. Look at the risks related to your product, processes, and resources in determining the extent of documented controls you need to have. The output of monitoring and measurement methods used within each QMS process provides useful performance indicators for determining the degree of conformity of product and QMS to requirements and whether the QMS has been effectively implemented and maintained. You must also establish methods and indicators to monitor and measure your QMS processes to demonstrate process capability to achieve planned results and identify opportunities to improve the process. Use your organization’s cross-functional knowledge of customer requirements, product, technology, manufacturing processes, etc., to determine process monitoring and measuring indicators and controls. Monitoring and measurement may be done manually or by automated means. Another way to identify useful methods and indicators is to review what problems could occur or have occurred within a particular process. Monitor and measure these occurrences and then analysis and evaluate the data to develop process controls to reduce or eliminate them. Problems or risks can occur with any of the variables in a process for e.g. materials, equipment, facility, methods, technology, personnel, computer hardware or software, etc. By using fishbone analysis or similar tools, you can develop very useful monitoring and measuring methods and process performance indicators. Correction refers to action taken to eliminate a detected nonconformity, i.e. the symptom. There must be a definite plan which determines what characteristics of product/service and process you will monitor and measure. When exactly will you monitor and measure those characteristics? How exactly will you analyze and evaluate the data you obtain from monitoring and measurement. Anywhen you are going to analyze and evaluate. The method you use should give you a valid result. You must also evaluate the performance and effectiveness of your QMS. You must keep a record of your monitoring, measurement, analysis, and evaluation. You must monitor your processes:
- First to determine and establish the capability of new processes to conform to requirements.
- And secondly, to monitor these processes over time to verify ongoing stability and capability to meet requirements.
- And thirdly to determine and achieve levels of continual improvement
To achieve planned results, control methods used for monitoring and measurement should focus on achieving the performance indicators we have identified for each QMS process. The monitoring and measurement techniques, sampling plans, acceptance criteria should be documented or referenced in your quality plan, or you could use a combination of specific practices, procedures, documents, and methods. Look at the risks and benefits in determining the extent of documented controls you need to have. However, the output of monitoring and measurement methods used within each QMS process provides useful performance indicators for determining the effective implementation and maintenance of QMS processes.
9.1.2 Customer Satisfaction
The organization should monitor customer perceptions of the degree to which their needs and expectations have been fulfilled and must determine the methods for obtaining, monitoring, and using this information. Some of the methods by which monitoring of customer perceptions can include customer surveys, customer feedback on delivered products or services, meetings with customers, market-share analysis, compliments, warranty claims, and dealer reports.
Customer satisfaction is still an important metric and now includes obtaining information on customer’s views; as well as having a structured approach to analyzing and evaluating all the information. Customers are primarily the end-users of your product, but also include intermediaries such as assemblers who may be internal or external and who integrate your product into theirs, and dealers and distributors who market and sell your product or the integrated product. You need to consider feedback from all these customers to determine whether or not you have met their specified and perceived requirements. Customer requirements may relate to the design, manufacture, delivery, servicing, and support of the product, QMS, communication, and financial requirements, etc. you must have controls to identify and meet these requirements Customer feedback or satisfaction is the first tool required by this standard to gauge the health of your QMS. This clause requires you to gather and analyze the information as to what extent you met these requirements, from the customer’s perspective. What is the customer’s evaluation of your performance with regard to their requirements? You must continually gather information about these activities, in a manner capable of being analyzed and evaluated to determine how well you performed them. There are all kinds of performance indicators for design, manufacture, delivery, service, and support, etc. Gather information on these indicators from both the customer as well from internal processes. There are many ways to monitor customer satisfaction feedback both positive as well as negative. These may include customer complaints, direct communications with customers, questionnaires and surveys, subcontracted collection and analysis of performance data, reports from consumer organizations, reports in various media, sectors, and industry studies. You are expected to have a process that defines your customer satisfaction indicators, frequency and method of data collection, summarization, review, and evaluation of data, actions to improve, timeline, responsibility, and follow-up. Many customers routinely provide feedback on some or all of the information indicated above. You must continuously review this customer feedback to ensure you maintain and improve your customer satisfaction rating. You must monitor trends in customer satisfaction indicators and use these as a baseline for continual improvement. You should consider both external as well as internal customer satisfaction. Note that every internal process is either a customer or a supplier of another process. You must identify and document the process addressing this clause as part of your QMS. For this process, you must also identify what specific documents, controls, and resources are needed. You could use a documented procedure or other combination of specific practices, procedures, documents, and methods. Look at the risks and benefits in determining the extent of documented controls you need to have. Performance indicators to measure the effectiveness of processes that control customer satisfaction may include improvement in customer feedback ratings, reduction in customer complaints, increase in the number of customers providing feedback, increase in feedback that leads to QMS, and product improvement opportunities.
9.1.3 Analysis and Evaluation
The organization should analyze and evaluate appropriate data and information arising from monitoring and measurement. Use the results of the analysis to evaluate the conformity of products and services, the degree of customer satisfaction, the performance and effectiveness of the quality management system. The organization must also evaluate if planning has been effectively implemented and the effectiveness of actions taken to address risks and opportunities. The performance of external providers and the need for improvements within the quality management system must also be evaluated. Methods to analyze data can include statistical techniques.
The Analysis and Evaluation requirements of the new standard are expressed in clause 9.1.3. The related requirements in ISO 9001:2008 are primarily in clause 8.4, Analysis of Data. A new requirement in clause 9.1.3 is to use the results of the analysis to evaluate if planning has been effectively implemented. Clause 8.4 of ISO 9001:2008 doesn’t mention the use of data analysis to evaluate planning. A new requirement in clause 9.1.3 is to evaluate the effectiveness of the actions taken to address risks and opportunities. ISO 9001:2008 does not mention risks or opportunities. Clause 8.4 in ISO 9001:2008 requires data to be analyzed to provide information on customer satisfaction. Clause 9.1.3 requires the analysis and evaluation to be used to evaluate the “degree” of customer satisfaction. Clause 8.4 in ISO 9001:2008 refers to analyzing data to provide information on conformity to product requirements. Clause 9.1.3 requires the results of the analysis to be used to evaluate the conformity of products and “services”.Clause 8.4 of ISO 9001:2008 requires data to be analyzed to “provide information” on “suppliers”. Clause 9.1.3 requires the results of the analysis to be used to evaluate the “performance” of “external providers”. An external provider is a supplier of products or services that is external to the organization. The provider could be a producer, distributor, retailer, or vendor.
You must collect and analyze QMS data that relate to the performance, effectiveness, and efficiency of products, services, QMS processes, production output, external provider (supplier) performance, use of resources, cost of poor quality, customer satisfaction, etc. Do year-over-year trend analysis to determine longer-term progress, identify opportunities for further improvement, or prioritize correction action for negative trends. Many organizations have wonderful systems for collecting data, but do a poor job in sorting, summarizing, and presenting this data for decision-making. You must sort and summarize the data you collect into things gone right and things that have gone wrong and present them separately. Management can then focus on continual improvement of things gone right and take corrective action on things gone wrong. Your process for data collection and analysis must address the type of data to be collected, how it should be sorted and classified, use of appropriate information systems and data gathering tools and techniques, assignment of responsibility and authority to review and act, competency, and training in the use of tools and data analysis, that the data is gathered, analyzed and acted upon on a timely basis. A summary of QMS performance data must be included in your periodic management review. Compare trends in quality and operational performance against your business plans, competitors, and industry benchmarks, where practical. Focus on key customer-related trends to prioritize prompt solutions to problems, determine longer-term planning for performance improvement, and to enhance customer satisfaction. Performance indicators to measure the effectiveness of processes for data collection and analysis may include a reduction in cycle time to gather and evaluate data, reduction in inaccurate and incomplete data, increase in improvement opportunities obtained from data analysis, etc.
9.3 Management Review
9.3.1 General
The Top Management of the organization should review the Organization’s QMS at planned intervals to ensure its continuing suitability, adequacy, effectiveness and it should be aligned with the strategic direction of the organization.
9.3.2 Management review inputs
Plan and carry out management review considering the status of actions from previous management reviews, changes in external and internal issues relevant to QMS, the adequacy of resources, opportunities for improvement, and the effectiveness of actions taken to address risks and opportunities as explained in clause 6.1. The organization must also consider information on quality performance and effectiveness, including trends in non-conformities and corrective actions, customer satisfaction and feedback from relevant interested parties, Monitoring and measurement results, Audit results, the extent to which quality objectives have been met, process performance, conformity of product and services, the performance of external providers
9.3.3 Management review outputs
Outputs from the management review must include decisions and actions related to opportunities for improvement, any need for changes to QMS, and resource needs. The organization should retain documented information as evidence of the results of management reviews.
The Management Review requirements of ISO 9001:2015 are now stated in clause 9.3, instead of clause 5.6 as in ISO 9001:2008. All of the clause 5.6 requirements of ISO 9001:2008 have been retained (except for the reference to preventive action). However, some have been reworded and relocated in a different sequence. For example, “follow-up actions from previous management reviews” was the fifth entry in 5.6.2 review inputs, and is now listed as the first consideration in 9.3.1 when planning and carrying out the management review: “status of actions from previous management reviews”. It makes sense to have “old” business covered first and then followed by “new” business. Clause 5.6.1 of ISO 9001:2008 required the review of any needed changes to the quality policy. The policy is no longer mentioned under Management Review. However, a similar and broader requirement is stated in 9.3.2: “any need for changes to the quality management system”. Clause 5.6.2 of ISO 9001:2008 requires that “information” be included for the listed review inputs. The ISO 9001:2015 standard changes the requirement to consider “information on the performance and effectiveness of QMS, including trends”, for the listed topics. The ISO 9001:2015 also requires you to plan and conduct your management reviews considering the changing business conditions and to align the reviews with your organizational strategies. An input in clause 5.6.2 of ISO 9001:2008 is to review the “status of preventive and corrective actions”. The related requirement in ISO 9001:2015 is to review “nonconformities and corrective actions”. A corrective action is a form of problem management, and preventive action is a form of risk management. Since the draft standard has added requirements on “risks and opportunities”, and is listed as a review topic, the reference to preventive action is no longer needed. Clause 5.6.1 of ISO 9001:2008 included “customer feedback” as a review input. This requirement has been revised in clause 9.3.1 to be “customer satisfaction and feedback from relevant interested parties”. A new requirement has been added as a review consideration, “The effectiveness of actions taken to address risks and opportunities.” It refers the reader to new clause 6.1, Actions to Address Risks and Opportunities, for more information. Clause 6.1 replaces the need for the old clause 8.5.3 on preventive action. Clause 5.6.1 of ISO 9001:2008 included “process performance and product conformity” as a review input. This requirement has been clarified as “process performance and conformity of products and services.”
The purpose of conducting management reviews of the QMS is to gauge the health of the QMS. The review must determine QMS suitability, adequacy, and effectiveness. Are the QMS resources and controls that were planned and implemented suitable and adequate for the QMS to be effective in achieving customer and regulatory requirements; and in achieving quality objectives? Are changes needed to improve products, processes, and use of resources? It would be logical to have a process for management review as it has specific requirements for management review inputs, value-adding review activities, and outputs. The process should address the frequency, schedule, quorum, and agenda for review meetings to be attended by top management. For the management review process itself to be effective, top management must plan the review of all agenda items with some regularity and take timely action to change or improve any part of it, including the quality policy and objectives. To avoid problems on frequency and scope of review, an effective way would be to incorporate QMS agenda items into regular monthly or quarterly operational meetings. Management review input should preferably be in summary form, showing QMS and operational performance measured against the business and quality plans, customer and regulatory objectives, and goals. Appropriate actions must result from such reviews. Review decisions and actions must relate to improving products and processes or even creating new ones, providing more resources or perhaps improving the efficiency of existing resources, improving QMS controls, objectives, improving overall QMS effectiveness and customer satisfaction. Responsibilities and timelines should accompany these decisions and actions. The performance of these actions must be followed up at subsequent management review meetings. Performance indicators to measure the effectiveness of the management review process could include – the achievement of quality objectives and improvement in customer satisfaction rating. You must identify and document the management review process as part of your QMS. You must also identify what specific documents are needed for effective planning, operation, and control of this process. These documents may include – documented information, review on, schedule, agenda and action forms, etc., combined with unwritten practices, procedures, and methods. Management review records must include topics discussed, decisions, responsibilities for corrective or improvement actions and related timelines, provision of resources, and follow-up actions from previous management reviews.
