Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties

Jun 21st, 2010 | By | Category: Auditing

Recently the U.S. Occupational Safety and Health Administration (OSHA) conducted a Process Safety Management (PSM) inspection at BP’s Husky Refinery near Toledo, Ohio.  The inspection resulted in 42 willful violations and approximately 3 million US dollars in proposed penalties.  One of the actions proposed by OSHA was that BP “audit a statistically significant number of pressure vessels, piping and instrument controls during the company’s PSM compliance audits.”  To my knowledge, this was the first time that statistically representative sampling has been proposed formally as part of any EH&S audit program.  To say nothing of what is truly meant by the “statistically significant” phrase.

This expectation by OSHA caused me to evaluate the following third-party auditing standards for statements related to sampling on audits:

  • The Board of Environmental, Health and Safety Auditing Certifications (BEAC) Standards, 2008
  • ISO 19011 Auditing Guidelines, 2002
  • Auditing Roundtable Standards, 1993
  • U.S. Environmental Protection Agency (USEPA) Auditing Policy, 1986, 2000
  • Institute of Internal Auditors Standards, 1997

A review of the above standards strongly suggests that only “appropriate sampling” or something comparable is the recommended practice.  The language in each standard that applies to the use of sampling techniques on EH&S audits is provided below.

This audit initiative by OSHA, a small part of a very significant enforcement action, could mean the beginnings of more rigorous sampling expectations on EH&S audits.  Or not.  My belief is that presently very few, if any, companies use formal statistically representative sampling on EH&S compliance audits.  Times could be changing; stay tuned.

BEAC Standards

“When conducting an audit, EH&S auditors shall use due care in examining and evaluating information they gather.  This information shall be sufficient, complete, relevant and useful to provide a sound basis for audit findings and recommendations.”  (Section III.8)

“Performance and Program Standards for the Professional Practice of Environmental, Health and Safety Auditing,” Board of Environmental, Health and Safety Auditor Certifications (BEAC), 2008

ISO 19011 Auditing Guidelines

“Audit evidence is verifiable. It is based on samples of the information available, since an audit is conducted during a finite period of time and with finite resources. The appropriate use of sampling is closely related to the confidence that can be placed in the audit conclusions.”  (Section 4(e))

“During the audit, information relevant to the audit objectives, scope and criteria, including information relating to interfaces between functions, activities and processes, should be collected by appropriate sampling and should be verified. Only information that is verifiable may be audit evidence. Audit evidence should be recorded.  The audit evidence is based on samples of the available information. Therefore there is an element of uncertainty in auditing, and those acting upon the audit conclusions should be aware of this uncertainty.”  (Section 6.5.4)

“Guidelines for quality and/or environmental management systems auditing,” International Organization for Standardization, ISO 19011:2002(E)

Auditing Roundtable Standards

“While on site, auditors must gather information necessary to fulfill the audit objectives. The information collected must be relevant, accurate, and sufficient to support findings, conclusions, and recommendations. Appropriate sampling schemes should be utilized in selecting samples.”  (Section II (C) (3))

“Minimum Criteria for the Conduct of EH&S Audits,” The Auditing Roundtable, 1993

U.S. EPA Auditing Policy

“A process which collects, analyzes, interprets and documents information sufficient to achieve audit objectives.”  (Appendix, Section V)

“Environmental Auditing Policy Statement,” US Environmental Protection Agency, July 9, 1986

Institute of Internal Auditors Standards

“Information should be sufficient, competent, relevant, and useful to provide a sound basis for audit findings and recommendations.

  1. Sufficient information is factual, adequate, and convincing so that a prudent, informed person would reach the same conclusions as the auditor.
  2. Competent information is reliable and the best attainable through the use of appropriate audit techniques.
  3. Relevant information supports audit findings and recommendations and is consistent with the objectives of the audit.
  4. Useful information helps the organization meet its goals.

Audit procedures, including the testing and sampling techniques employed, should be selected in advance, where practicable, and expanded or altered if circumstances warrant.”  (Sections 420.2 and 420.3)

 Standards for the Professional Practice of Internal Auditing, The Institute of Internal Auditors, 1997

About the Author

Lawrence B. Cahill, CPEA, is a Technical Director at Environmental Resources Management in Exton, Pennsylvania, U.S.A. 

Mr. Cahill has over 30 years of professional EH&S experience with industry and consulting.  He is the principal author of the widely used text, Environmental, Health and Safety Audits, published by Government Institutes, Inc. and now in its 8th Edition.  He contributed four chapters in the 1995 book Auditing for Environmental Quality Leadership, published by John Wiley & Sons, Inc.  Mr. Cahill has published over 50 articles and has been quoted in numerous publications including the New York Times and the Wall Street Journal.

Photograph:  Blue Butterfly by Pigi Vigi, Tartu, Eesti, Estonia.

Return to the EHS Journal Home Page.

Tags: , , , , , ,

8 Comments to “Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties”

  1. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties (Cahill) […]

  2. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties (Cahill) […]

  3. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties  […]

  4. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties […]

  5. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties […]

  6. […] Statistically Representative Sampling on EH&S Audits: Expectations Established by Third Parties […]

  7. Randy Putnam says:

    There are several sources for statistical sampling. A very simple one is Det Norske Veritas’s Safety Management Systen Audits. They provide a table for interviewing employees (statistical sample) based on the total number of employees. Simple and effective. It maxes out after a certain number of employees. Six Sigma and other quality programs also provide means to determine what a statistically significant sample is. Generally, the parameters are: how confident do you want to be in the results, how large is the population (i.e. how many tanks are there) and what percentage are expected to fail.
    If you sample say 10 randomly selected items and there are no “failures” you may determine that is a statistically significant result.

  8. Joel Olener says:

    Actually, this is not the first time this has happened with OSHA under the PSM standard. There was a case recently where OSHA had cited a company for not using statistical sampling for the audit. After reviewing the PSM standard the only place where I could find any reference to “sampling” was in the NON-MANDATORY (emphasis added) Appendix C. I am quoting it here:

    “An audit is a technique used to gather sufficient facts and information, including statistical information, to verify compliance with standards. Auditors should select as part of their preplanning a sample size sufficient to give a degree of confidence that the audit reflects the level of compliance with the standard. The audit team, through this systematic analysis, should document areas which require corrective action as well as those areas where the process safety management system is effective and working in an effective manner. This provides a record of the audit procedures and findings, and serves as a baseline of operation data for future audits. It will assist future auditors in determining changes or trends from previous audits.”

    I apologize in advance for the following convoluted sentences: I was curious and called OSHA to ask if non-mandatory means it isn’t mandatory, or if it is required. The OSHA person I spoke to said that the only thing that was mandatory was the regulation itself, but would not (or could not) say if non-mandatory requirements were required.

    Based on my experience, sample size for audits is a judgment call. For example, if there are 10 vessels on a site, an audit of their vessel inspection program would probably include 5 of them. If the site had a hundred, then 5 is not sufficient, but 10 or 15 might be. If the inspection universe was 1000 vessels, I don’t think 100 records at random might be reasonable, but 20 or 30 targeted reviews might be. In any case, if the first few inspection records indicate non-compliance with the requirements for inspection, then more records would be looked at.

    On the other hand, a large refinery might have a couple of thousand Hot Work records every year. I submit that looking at a 10 percent sample size for the three year audit period would be cumbersome and nonproductive.

    While I do believe it would be nice to do formal statistical sampling (such as one might do for product quality control) in an audit, it is not required nor in many cases warranted.

    Comments and feedback welcome either publicly or to me privately at joel.olener@pssillc.com.

Leave a Comment