Most fields require very high probabilities of safety, even when the stakes are much lower than the extinction of humanity.[1] Here are a few examples with risk thresholds expressed in Log Odds (base 10).
Intro to Log Odds
Instead of writing probabilities like 0.9999 which is hard to read, I'll use log odds.
LogOdds(p)=log10(p¬p)=log10(p1−p)
So LogOdds(0.9999)≈4
Conversion Trick: rounding LogOdds towards zero[2] roughly corresponds to the number of nines/zeros after the dot in the probability. Conveniently, the complementary probability in log space is just negative. LogOdds(1−p)=−LogOdds(p)
regulation asking for fake low probabilities backed up by clever experiments
What I'd like to convey:
Most certainty/risk standards have thresholds set such that failures basically never happen, and those that do probably didn't follow the standard (though causality is hard to establish)
Abundant caution is more important than perfectly calibrated risk estimates on catastrophe
Risk should be "as low as reasonably practicable"
Many people are too comfortable with probabilities of human extinction far in excess of the guidelines regulating individual human lives.
There is no looming existential threat against which superintelligence is our only hope, so when you're gambling with everyone's lives and you don't get second chances, your risk tolerance should be considerably lower than for eg a skyscraper collapsing.
While I'm grateful for LLM research assistance for this report, I'd happily trade the time savings for safety. Please face God and walk backwards away from the cliff edge.
If useful to this report, I'm open to adding:
more fields with explicit probability thresholds
studies on safety outcomes downstream of the guidelines
Minimum Design Loads and Associated Criteria for Buildings and Other Structures published by the American Society of Civil Engineers (ASCE) and its Structural Engineering Institute (SEI).
US Nuclear Regulatory Commission (USNRC) Safety Goal Policy established Quantitative Health Objectives (QHOs) that plant-attributable risk of cancer/early fatalities should be less than 0.1% of respective background risk. OECD/NEA evaluated baselines in 1980s US to extrapolate absolute risks, which continue to be referenced and haven't been rebaselined.
Most fields require very high probabilities of safety, even when the stakes are much lower than the extinction of humanity.[1] Here are a few examples with risk thresholds expressed in Log Odds (base 10).
Intro to Log Odds
Instead of writing probabilities like 0.9999 which is hard to read, I'll use log odds.
LogOdds(p)=log10(p¬p)=log10(p1−p)
So LogOdds(0.9999)≈4
Conversion Trick: rounding LogOdds towards zero[2] roughly corresponds to the number of nines/zeros after the dot in the probability. Conveniently, the complementary probability in log space is just negative. LogOdds(1−p)=−LogOdds(p)
Civil Engineering
United States
General Structural Design (ASCE/SEI 7-22)[3]
Target Reliability Table (excluding earthquake, tsunami, and extraordinary events) Annual probability of various types of failure.
S= sudden, W= widespread progression of damage
Bridges (AASHTO LRFD)
Lifetime Reliability Target (β=3.5)
LogOdds: -3.63 lifetime reliability (-5.51 annual, for a typical 75 year service life)
Dams (Bureau of Reclamation Public Protection Guidelines)
Piecewise log-log fatality-probability guidelines. (-1 slope represents constant Annualized Life Loss)
ALARP = As Low As Reasonably Practicable
"ALARP is now considered to apply anywhere on the risk portrayal chart"
Case Study: Citicorp Center[8]
1/16 annual chance of collapse was unacceptable
LogOdds: -1.18
Europe
Building Minimum Reliability[9] (EN 1990)
Netherlands - Fundamentals of Flood Protection (2017)
Log-log death-frequency target for 2050
China
Building Annual Reliability Index (GB 50153‑2008)
Highway Bridges, Culverts, and Tunnels (JTG 2120‑2020)
Automotive
International
Random Hardware Failures (ISO 26262)
Aerospace
United States
Airplane Failure Probability-Severity per flight-hour (AC 25.1309-1)
Minor
(-3 to -5)
Major
(-5 to -7)
Hazardous
(-7 to -9)
Catastrophic
(less than -9)
Airplane
(reduction in functional capabilities or safety margins)
in workload
incapacitation
discomfort
distress,
possibly
including
injuries
injury to a small
number of
persons
fatalities
Space (14 CFR 450.101)
Risk per launch to:
Nuclear
OECD (NEA MDEP)
United Kingdom
Carcinogens
International
Threshold of Toxicological Concern (ICH M7 R1)[20]
Excess lifetime risk at 1.5μgday: -5
United States
EPA: Lifetime Risk from Air Pollutants (NESHAP)
Living <50km from plant: -4
General Public: -6
Electronics
International
Safety Integrity Levels (IEC 61508)
Max Probability of Failure on Demand
Max Probability of Dangerous Failure per Hour
Physics
Particle Physics & Gravitational Waves
5 sigma statistical significance (usually single-tailed): 6.54
My Ideological Conclusion (optional)
Things I don't want:
What I'd like to convey:
If useful to this report, I'm open to adding:
Inspired by Rob Miles on Doom Debates
Round positive values down. Round negative values up.
trunc()in programming.Minimum Design Loads and Associated Criteria for Buildings and Other Structures published by the American Society of Civil Engineers (ASCE) and its Structural Engineering Institute (SEI).
Storage Sheds, Temporary Structures, Low Occupancy, Low-Risk Structures
Homes, Businesses, Most Structures, Standard Occupancy Dwellings
Schools, Gathering Places, Public Utilities, High Occupancy Gathering Halls
Hospitals, Emergency Utilities, Sensitive Occupancy, High Risk Structures
Wikipedia, Veritasium
LogOdds(β)=log10(Φ(−β)1−Φ(−β)) where Φ is the CDF of the standard normal distribution
Agricultural buildings where people do not normally enter. (e.g. storage buildings, greenhouses)
Public buildings where consequences of failure are medium. (e.g. residential, office buildings)
Public buildings where consequences of failure are high. (e.g. grandstands, concert halls)
Small or temporary storage buildings
Ordinary residential buildings and office buildings
Large-scale public buildings
Rural tunnels ≤1000 m, construction access tunnels
2-lane urban tunnels, rural tunnels >1000 m
Expressways, multi-arch tunnels, tunnels ≥3 lanes or ≥3000 m
US Nuclear Regulatory Commission (USNRC) Safety Goal Policy established Quantitative Health Objectives (QHOs) that plant-attributable risk of cancer/early fatalities should be less than 0.1% of respective background risk. OECD/NEA evaluated baselines in 1980s US to extrapolate absolute risks, which continue to be referenced and haven't been rebaselined.
International Council for Harmonization (ICH) of Technical Requirements for Pharmaceuticals for Human Use: Assessment and Control of DNA Reactive (Mutagenic) Impurities in Pharmaceuticals To Limit Potential Carcinogenic Risk - M7(R1)
deployed ≤1 time per year
deployed >1 time per year