Fiber is defined as the parts of plants that we can eat but not digest. We're always told to eat more of it. Why should we eat things we can't digest?

There are many papers to point to. Fiber consumption is strongly associated with less cancer, less heart disease, and less death (recent meta-analysis here). Most of these findings come from observational studies, where people report their diets to researchers. Studies in which fiber consumption is experimentally controlled tend to find fewer benefits, so people who eat more fiber may already be healthier for other reasons.

Sidestepping the causal question, we could still ask: what are the plausible mechanisms? How does fiber do this? 

Below is a list of potential benefits from an authoritative-seeming (~1,000 cites) paper in the journal Nutrients. I'm struck by how long the list is. I'm still inclined to follow the conventional advice--it feels right and if something was severely harmful about fiber we'd know from the observational data. But I do wonder: with such diverse and complex impacts on the body, could there be various negative effects too? 

         Reasons fiber could decrease cancer

1. Dietary fiber (DF) is “fermented to produce short chain fatty acids” in the large intestine, “which have anti-carcinogenic properties”
2. Since “DF increases fecal bulking and viscosity, there is less contact time between potential carcinogens and mucosal cells”
3. “DF increases the binding between bile acids and carcinogens”
4. DF increases antioxidants
5. DF decreases estrogen (which could cause cancer)
   
   Reasons fiber could decrease heart disease
    
6. “[S]oluble fibers have been shown to increase the rate of bile excretion therefore reducing serum total and LDL cholesterol”
7. “[S]hort chain fatty acid production [(see point 1)], specifically propionate, has been shown to inhibit cholesterol synthesis”
8. DF “regulate[s] energy intake [I believe through a satiety effect; you feel more full] thus enhancing weight loss or maintenance of a healthier body weight”
9.  DF “decrease[s] pro-inflammatory cytokines such as interleukin-18 which may have an effect on plaque stability”
   
   Reasons fiber could decrease diabetes
   
   The authors focus on cancer and heart disease protection because these effects have been approved by the FDA. The paper also mentions this protection against diabetes:
    
10. “Dietary fibers have been shown to result in decreased blood glucose excursions and attenuated insulin responses. This may be due to either a delayed or decreased intestinal absorption.”
    
    Reasons fiber could…be a scrub brush
     
11. "Fiber cleans your colon, acting like a scrub brush. The scrub-brush effect of fiber helps clean out bacteria and other buildup in your intestines, and reduces your risk for colon cancer." This is not from the paper, it's from this UCSF page. 

There are even more reasons in other papers. For example, this one mentions benefits for the immune system.