An Approach to Calibrating Relative Risks on EHS Audits

Mar 12th, 2016 | By | Category: Featured Articles

EHS Journal - Draw Serie by Miguel Ugalde

“It is better to know some of the questions than all of the answers.” James Thurber

Introduction

In recent years there has been considerable discussion of establishing a more risk-based focus on environmental, health, and safety (EHS) audits. The concern is that audit results had devolved over time and several audit cycles into an inventory of minor infractions and administrative regulatory deficiencies. Line managers were not so sure that the effort put into the audits was identifying true and important risks and/or that maybe the sites were asymptotically approaching full compliance. On the other hand, EHS incidents experienced by BP, TEPCO, Duke Energy, Volkswagen, and other major corporations suggested that there were still major risks out there that needed to be anticipated or identified.

There are any number of ways to re-engineer an audit program to better focus on risks and achieve more value from the process. Examples might include:

  • Increase the frequency of audits for inherently higher-risk facilities
  • Increase the frequency of facility audits in highly regulated and/or high-profile settings
  • Assure that non-traditional operations and remote locations are included as part of the audit universe and that these are audited at the appropriate depth and frequency, even if by an independent, third-party
  • Conduct single-focus, deep-dive audits on specialized, high-risk topics such as hazardous waste management
  • Require that root-cause analysis be conducted by auditors on perceived high-risk observations
  • Add subject matter experts to audit teams on high-risk, specialized topics such as the mechanical integrity aspects of process safety management

One additional approach often implemented to help managers and executives better understand and interpret the results of facility audits is to classify EHS audit findings by level of risk: high, medium, or low; significant, major, or minor, and so forth. The goal is to help to separate “the wheat from the chaff”, so to speak.  Yet, even with articulated definitions of risk levels, it is often difficult for even the most experienced audit professional to anticipate the relative consequences and potential impacts of different findings and observations.  Why is this the case?  Because like many things in life – it’s complicated.  This article focuses on the reason for these complications.

 

The Challenge of Assessing Relative Risks

Over many years of auditing, I have had to work with audit teams to determine risk levels for individual audit findings. This has not always been a straightforward experience as two trained audit professionals can interpret the very same set of facts in widely divergent ways.  Ten examples of some of the more troublesome comparisons I’ve experienced are listed in Tables 1 and 2.  Take a look, and see what you think.  Your task is to select the higher risk observation in each case.  And think about the choices as Jack Welch, the former Chairman of General Electric, did with his managers when evaluating staff performance.  He required them to force-rank all staff, and the bottom 10% were vulnerable to re-assignment or even termination.  Managers were not allowed to conclude that all their staff were above average.

So, in line with Mr. Welch’s philosophy, force-rank each of the scenarios provided below by choosing either Scenario A or Scenario B. You are not allowed to conclude that the risks are roughly equal.  You must choose one over the other.  When choosing try to keep in mind not just the observation itself but what the observation might say about the underlying controls or management systems in place or not.  This might say something about whether that very same issue might be observed yet again on the subsequent audit conducted three years later.  After evaluating the tenth scenario, it should become evident just how difficult it is to indeed assign risk on EHS audits.  But that is exactly what auditors are expected to do.

 

Table 1: Making EHS Risk Choices for Environmental Topics

EHS Journal - Cahill Table Env

 

Table 2: Making EHS Risk Choices for Health & Safety Topics

I can only hope that you were as perplexed as I was with choosing the higher risk finding in each of the scenarios posed above. And this has not been a “trick exercise” where I am about to provide the magic answer.  There is no magic answer to these choices.

 

Calibrating Relative Risks

One useful exercise would be to have a group of auditors in an organization each make the choices for the 10 scenarios given above. Then the overall percentage distributions could be calculated and entered into Table 3, Evaluating Risk Choices.  For the cases where the result is that either Choice A or Choice B was made by close to 100% of those polled, that would suggest that there is a commonality of thinking amongst colleagues about relative risks posed by EHS issues.  Where the results are closer to 50/50, then some organizational calibration might be required.  Auditors working for the same organization should view risks pretty much the same way.  A 50/50 split would suggest that they do not hold the same risk perceptions.  In these cases, the program is at risk of being viewed as being inconsistently applied.  And that is never a good thing for an audit program.

Table 3: Score Sheet

EHS Journal - Cahill Table 3 Scoresheet

 

Closure

It is indeed a good thing that EHS audit programs are generally moving towards more risk-based approaches. Line managers and executives both really need to know the difference between real potential problems and administrative glitches.  Yet assigning and evaluating risks on individual findings can be a challenging exercise.  This is particularly true on the safety side of the business where even the most mundane of deficiencies can be viewed as life-threatening under a worst-worst case scenario.  Training and calibration among a company’s professional auditors on relative risks posed by EHS findings of deficiency can help to lead the way to making any audit program more useful to the organization.

 

References

  1. Cahill, L.B. and R.J. Costello, “EHS Audits-Have We Lost Our Way, Part III,” EHS Journal On-Line, March 12, 2013.
  2. Cahill, L.B. and R.J. Costello, “EHS Audits-Have We Lost Our Way? A Sequel,” EHS Journal On-Line, August 13, 2011.
  3. Cahill, L.B., “EH&S Audits – Have We Lost Our Way?” EHS Journal On-Line, July 11, 2010

About the Author

Lawrence B. Cahill, CPEA (Master Certification) is a Technical Director with Environmental Resources Management and has over 35 years of professional EHS experience with industry and consulting.  He is the editor and principal author of the widely used text, Environmental, Health and Safety Audits, 9th Edition and its 2015 follow-up text EHS Audits: A Compendium of Thoughts and Trends, both published by Bernan Press.  He has published over 70 articles and has been quoted in numerous publications including the New York Times and the Wall Street Journal.  Mr. Cahill has worked in over 25 countries during his career. He holds a B.S. in Mechanical Engineering from Northeastern University where he was elected to Pi Tau Sigma, the International Mechanical Engineering Honor Society.  He also holds an M.S. in Environmental Health Engineering from the McCormick School of Engineering and Applied Science of Northwestern University, and an MBA from the Wharton School of the University of Pennsylvania.  He is a Certified Professional Environmental Auditor, Master Certification.

Photograph: Drawing Serie by Miguel Ugalde.

 

Return to the EHS Journal Home Page

Tags: , , , ,

3 Comments to “An Approach to Calibrating Relative Risks on EHS Audits”

  1. Rob Peddle says:

    Excellent examples of the real difficluty of getting consistency into auditor (and management) interpretation of evidence – what is the real level of risk this indicates?. Of course, the real risk level is only identified when bringing together a number of pieces of evidence, rather than just from consid3eration of a single item, so making decisions based on any single items does not necessarily give the right answer – and may often be completely wrong! This also oftern leads to differences of opinion, both between auditors and with our client management as well!

    The reality is that there will be a range of compliance, documentary, behavioural and other evidence that need to be considered before a clear picture of risk is understood and this is the real competence that is needed. Trying to address this need by using just traditional auditing tools and techniques and expecting the auditors to somehow ‘fill the gap’ is unreasonable and will always create a significant risk in itself. If we are to fully embrace the risk-based audit requirement and provide the service expected, we need to add to our toolkit and accept the validity of other forms of eveidence and methods of ts collection and analysis. If we try to ‘do better’ with what we have always used, with a focus on auditor training only, we will fail, both ourselves and our customers. Such tools are already there if we want to accept them!

  2. Pranav Sinha says:

    Larry,
    The challenge you present above is one that I have faced over and over again, both as an audit team member and as the Project Manager for multi-site (at times multi-country) audits programs. Also, as you correctly stated there are no simple answers nor too many clear rules to follow. This single deficiency can significantly lower the value of the output of an audit program, even as the attempt to bring consistency across multiple reports increases the effort required manifold.
    Pranav

  3. Peter Temesvary says:

    Thanks Larry,
    A very thought provoking article! As managers of audit teams we could pay a lot more attention to this, and incorporate this type of excercise into trainings. Consistency among auditors can indeed be a weak link in an audit program, especially in this age of global audit programs and teams.
    Peter Temesvary

Leave a Comment