Most fields require very high probabilities of safety, even when the stakes are much lower than the extinction of humanity.[1] Here are a few examples with risk thresholds expressed in Log Odds (base 10).
Intro to Log Odds
Instead of writing probabilities like 0.9999 which is hard to read, I'll use log odds.
So LogOdds
Conversion Trick: rounding LogOdds towards zero[2] roughly corresponds to the number of nines/zeros after the dot in the probability. Conveniently, the complementary probability in log space is just negative. LogOddsLogOdds(p)
Civil Engineering
United States
General Structural Design (ASCE/SEI 7-22)[3]
Target Reliability Table (excluding earthquake, tsunami, and extraordinary events) Annual probability of various types of failure.
sudden, widespread progression of damage
| Types of Failure | Risk Categories | |||
| I[4] | II[5] | III[6] | IV[7] | |
| -3.90 | -4.52 | -4.90 | -5.30 | |
| -4.52 | -5.30 | -5.70 | -6.15 | |
| -5.30 | -6.15 | -6.60 | -7.00 | |
Bridges (AASHTO LRFD)
Lifetime Reliability Target ()
LogOdds: -3.63 lifetime reliability (-5.51 annual, for a typical 75 year service life)
Dams (Bureau of Reclamation Public Protection Guidelines)
Piecewise log-log fatality-probability guidelines. (-1 slope represents constant Annualized Life Loss)
ALARP = As Low As Reasonably Practicable
"ALARP is now considered to apply anywhere on the risk portrayal chart"
Case Study: Citicorp Center[8]
1/16 annual chance of collapse was unacceptable
LogOdds: -1.18
Europe
Building Minimum Reliability[9] (EN 1990)
| Reference Periods | Reliability Classes | ||
| RC1[10] | RC2[11] | RC3[12] | |
| 1 year | -4.87 | -5.89 | -7.00 |
| 50 years | -3.32 | -4.14 | -5.07 |
Netherlands - Fundamentals of Flood Protection (2017)
Log-log death-frequency target for 2050
China
Building Annual Reliability Index (GB 50153‑2008)
| Type of failure | Safety Class | ||
| Class III[13] | Class II[14] | Class I[15] | |
| Ductile (gradual) | -2.46 | -3.16 | -3.97 |
| Brittle (sudden) | -3.16 | -3.97 | -4.87 |
Highway Bridges, Culverts, and Tunnels (JTG 2120‑2020)
| Type of failure | Safety Class | ||
| Class III[16] | Class II[17] | Class I[18] | |
| Ductile (gradual) | -3.97 | -4.87 | -5.89 |
| Brittle (sudden) | -4.87 | -5.89 | -7.00 |
Automotive
International
Random Hardware Failures (ISO 26262)
| ASIL | Examples | Target Failure Rate per Hour |
| B | Headlights, Brake Lights | -7 |
| C | Cruise Control, Rear Wheel Brakes | -7 |
| D | All Wheel Brakes | -8 |
Aerospace
United States
Airplane Failure Probability-Severity per flight-hour (AC 25.1309-1)
| Effects | Severity (LogOdds Range) | |||
Minor (-3 to -5) | Major (-5 to -7) | Hazardous (-7 to -9) | Catastrophic (less than -9) | |
Airplane (reduction in functional capabilities or safety margins) | Slight | Significant | Large | Normally hull loss |
| Crew | Slight increase in workload | A physical discomfort or significant increase in workload or in conditions impairing the efficiency of the flight-crew | Physical distress or excessive workload such that flight-crew cannot be relied upon to perform their tasks accurately or completely | Fatalities or incapacitation |
| Passengers | Physical discomfort | Physical distress, possibly including injuries | Serious or fatal injury to a small number of persons | Multiple fatalities |
Space (14 CFR 450.101)
| Risk per launch to | Overall | Most Endangered Individual |
| Operations Personnel | -3.7 | -5 |
| The Public | -4 | -6 |
Risk per launch to:
- Other Aircraft: -6
- Critical Assets (national interests on the ground): -3
- Critical Payload (national interests on the rocket): -4
Nuclear
OECD (NEA MDEP)
| Country/Org | Page | Event Frequency/Year | |
| Core Damage | Large Release | ||
| INSAG | 11 | -5 | -6 |
| United States | 14[19] | -4 | -5 |
| Russia | 26 | -5 | -7 |
| France | 17 | -5 | -7 |
| Japan | 23 | -4 | -5 |
| Finland | 18 | -5 | -6.3 |
| Canada | 22 | -5 | -6 |
United Kingdom
| Initiating Fault Frequency | Basic Safety (mSv) | ||
| Limit | Objective | ||
| On-site | > -3 | 20 | 0.1 |
| -3 to -4 | 200 | 0.1 | |
| < -4 | 500 | 0.1 | |
| Off-site | > -3 | 1 | 0.01 |
| -3 to -4 | 10 | 0.01 | |
| < -4 | 100 | 0.01 | |
Carcinogens
International
Threshold of Toxicological Concern (ICH M7 R1)[20]
Excess lifetime risk at : -5
United States
EPA: Lifetime Risk from Air Pollutants (NESHAP)
Living <50km from plant: -4
General Public: -6
Electronics
International
Safety Integrity Levels (IEC 61508)
| SIL | Low Demand Mode[21] Max Probability of Failure on Demand | High Demand or Continuous Mode[22] Max Probability of Dangerous Failure per Hour |
| 1 | -1 | -5 |
| 2 | -2 | -6 |
| 3 | -3 | -7 |
| 4 | -4 | -8 |
Physics
Particle Physics & Gravitational Waves
5 sigma statistical significance (usually single-tailed): 6.54
My Ideological Conclusion (optional)
Things I don't want:
- clever arguments from AI labs for P(doom)
- regulation asking for fake low probabilities backed up by clever experiments
What I'd like to convey:
- Most certainty/risk standards have thresholds set such that failures basically never happen, and those that do probably didn't follow the standard (though causality is hard to establish)
- Abundant caution is more important than perfectly calibrated risk estimates on catastrophe
- Risk should be "as low as reasonably practicable"
- Many people are too comfortable with probabilities of human extinction far in excess of the guidelines regulating individual human lives.
- There is no looming existential threat against which superintelligence is our only hope, so when you're gambling with everyone's lives and you don't get second chances, your risk tolerance should be considerably lower than for eg a skyscraper collapsing.
- While I'm grateful for LLM research assistance for this report, I'd happily trade the time savings for safety. Please face God and walk backwards away from the cliff edge.
If useful to this report, I'm open to adding:
- more fields with explicit probability thresholds
- studies on safety outcomes downstream of the guidelines
- ^
Inspired by Rob Miles on Doom Debates
- ^
Round positive values down. Round negative values up.
trunc()in programming. - ^
Minimum Design Loads and Associated Criteria for Buildings and Other Structures published by the American Society of Civil Engineers (ASCE) and its Structural Engineering Institute (SEI).
- ^
Storage Sheds, Temporary Structures, Low Occupancy, Low-Risk Structures
- ^
Homes, Businesses, Most Structures, Standard Occupancy Dwellings
- ^
Schools, Gathering Places, Public Utilities, High Occupancy Gathering Halls
- ^
Hospitals, Emergency Utilities, Sensitive Occupancy, High Risk Structures
- ^
- ^
where is the CDF of the standard normal distribution
- ^
Agricultural buildings where people do not normally enter. (e.g. storage buildings, greenhouses)
- ^
Public buildings where consequences of failure are medium. (e.g. residential, office buildings)
- ^
Public buildings where consequences of failure are high. (e.g. grandstands, concert halls)
- ^
Small or temporary storage buildings
- ^
Ordinary residential buildings and office buildings
- ^
Large-scale public buildings
- ^
Rural tunnels ≤1000 m, construction access tunnels
- ^
2-lane urban tunnels, rural tunnels >1000 m
- ^
Expressways, multi-arch tunnels, tunnels ≥3 lanes or ≥3000 m
- ^
US Nuclear Regulatory Commission (USNRC) Safety Goal Policy established Quantitative Health Objectives (QHOs) that plant-attributable risk of cancer/early fatalities should be less than 0.1% of respective background risk. OECD/NEA evaluated baselines in 1980s US to extrapolate absolute risks, which continue to be referenced and haven't been rebaselined.
- ^
International Council for Harmonization (ICH) of Technical Requirements for Pharmaceuticals for Human Use: Assessment and Control of DNA Reactive (Mutagenic) Impurities in Pharmaceuticals To Limit Potential Carcinogenic Risk - M7(R1)
- ^
deployed ≤1 time per year
- ^
deployed >1 time per year