How to end credentialism

[-]Gordon Seidoh Worley7mo105

We can design a better educational institution if we separate assessment from teaching. We can do better than you propose if assessment is fully decoupled from teaching. MIT wouldn't hand out degrees; some other body would. MIT's role would be to educate people to be able to pass those assessments to the extent anyone cared about performance on those assessments.

Of course there's a bunch of ways I expect such a design to fail, but if the goal is education, then this seems like a more efficient way to do it.

[-]Yair Halberstadt7mo30

As stated in the OP, I expect this to be the end result of the regulations I suggest. The advantage of this approach is that for now MIT can carry on doing it's thing instead of forcing a hard switch over where you stop it being able to assess is students without yet having an equivalently respected replacement.

[-]Mis-Understandings7mo52

Credentialism is good because the limiting factor on employment is trust, not talent for most credential requiring positions (white collar, buisness and engineering work).

Universities are bad at teaching skills, but generate trust and social capital.

Trust that allows the system to underwrite new white collar workers to do things that might lose buisnesses lots of money is important and expensive.

Consequently you get credential requirements, because there is no test other than years of being in social systems that can tell you that a person has the ability to go 4 years without crashing out (which is the key skill).

Additionally, going to university has become a class signifier, and all classes wish they were bigger and more prominent.

The alternative to credentialism is selection, or real meritocracy.

The alternative to credentialism is not selection, it is hiring your buddies, hiring by visible factors, and hiring randomly. Most business are not that guy that they can run a competitive selective process (THOSE ARE REALLY EXPENSIVE).

"universities provide to employers is the ability to confirm you are clever, driven, and have relevant skills" is false. They provide that you are a member of the professional class that is not going to do stupid things that lose money/generate risk.

Fundamentally, this misunderstands the purpose of the degree to the hiring bureaucracy, and the political economy behind it.

[-]Yair Halberstadt7mo20

And how exactly are universities a good signifier of that? Note I took an external degree from the university of London, which even if universities were a good signifier of that, this one definitely wasn't, and it did not in any way impact my ability to get a job. Noone cared.

[-]Mis-Understandings7mo50

Noone cared.

You don't know what questions they did not ask you, and the assumptions of shared cultural background that they made because they saw that. They would not tell you. (unless you have comparisons to job searching before getting the degree).

Fundamentally, this is the expected phenomenology, since people do not tend to notice sources of your own status.

[-]Quinn7mo3-1

One thing I like about this is making the actual difficulty deltas between colleges more felt/legible/concrete (by anyone who takes the exams). What I might do in your system at my IQ level (which is pretty high outside of EA but pretty mediocre inside EA) is knock out a degree at an easy university to get warmed up then study for years for a degree at a hard school^[1].

In real life, I can download or audit courses from whatever university I want, but I don't know what the grading curve is, so when 5/6 exercises are too hard I don't know if that's because I'm dumb or if 1/6 is B+ level performance. This is a way that the current system underserves a credential-indifferent autodidact. It's really hard to know how difficult a course is supposed to be when you're isolated from the local conditions that make up the grading curve!

Another thing I like about your system is tutoring markets separated from assessment companies. Why is it that we bundle gatekeeping/assessment with preparation? Unbundling might help maintain objective standards, get rid of problems that look like "the professor feels too much affection for the student to fail them".

This is all completely separate for why your proposal is a hard social problem / a complete nonstarter, which is that I don't think the system is "broken" right now. There's an idea you might pick up if you read the smarter leftists, which is that credentialism especially at elite levels preserves privilege and status as a first class use case. This is not completely false today, not least because the further you go back in time in western universities the truer it is.

my prior, 15 years ago, looked like "stanford has a boating scholarship, so obviously selectivity is a wealth/status thing and not reflective of scholarship or rigor", so the fact that I now believe "more selective colleges have harder coursework" means I've seen a lot of evidence. It pains me, believe me, but reality doesn't care :) ↩︎

[-]Ariel7mo1-1

For [1], could you point at some evidence, if you have any on hand? My impression from TAing STEM at an Ivy League school is that homework load and the standards for its grading (as with the exams) is very light, compared to what I remember from my previous experience in a foreign state university.

It wasn't at all what I expected, and shaped (among other signals of implied preference by the university) my view that the main services the university offers to its current and former students are networking opportunities and a signal of prestige.

[-]Quinn7mo20

I don't know what legible/transferable evidence would be. I've audited a lot of courses at a lot of different universities. Anecdote, sorry.

[-]Viliam7mo20

If we tried to make universities examine non-students, the easiest way for the universities to cheat would be to teach literal passwords to their students, and then fail everyone who doesn't know the password. Then they could demonstrate that even in a fair competition only their students are able to pass the exams.

With some cleverness you could insert arbitrary passwords even into hard sciences, for example by asking "what was Einstein's most important contribution to physics?" Even if you knew everything about physics, and everything about Einstein, it wouldn't help you figure out what was his most important contribution in the teacher's opinion. (Could be any of them. Could even be something meta.)

Or you could take a set of arbitrary details (something so specific that it is virtually impossible for a human to memorize all things in the same category of relevance) and keep repeating the selected details to your students every year. Ask those details in exams every year; then ask them again on the final exam.

And with soft sciences... basically everything already is like this. (Or am I wrong? There is a way to test this empirically: take students from a few universities and make them take exams at a different university that teaches nominally the same subject; preferably in a different country.)

[-]Yair Halberstadt7mo40

Note this would be illegal (under the term that requires you to provide all resources specifically relating to the exam as opposed to the subject in general to external students).

It would also cause students to appeal, and the statistics would be obvious enough that the appeals committee would investigate, ask for the mark scheme, and would quickly find on the students side because the paper clearly contains arbitrary details designed to do this.

Note there's a lot of things that are like this in law, where people could in theory cheat, possibly even within the letter of the law, but they don't because the courts throw the book at them when they do.

[-]frontier647mo10

Instead of doing this why not just, not require credentialism? Let the free market regulate doctors. Eliminate laws which put onerous credential requirements on professions where there's high demand and low supply.

[-]Yair Halberstadt7mo40

How? There's no law requiring software developers to have a degree, but employers often still only accept people who do.

[-]frontier647mo10

Employers who poorly select employees will most likely be out-competed by employers who make better decisions when selecting employees. We already see this in hiring for software engineers where many employers will accept a bachelors degree or other experience which demonstrates coding skill.

And for software engineering, at least when I went to school, you still had to be able to program at least a little bit to get a degree.

The majority of the downside from credentialism comes from fields where it’s literally illegal to work if you don’t have the right college degree

[-]Yair Halberstadt7mo20

Why isn't the fact software developers spend 3 years not learning all that much (far less than they would in 6 months on the job) not a problem?

[-]frontier647mo10

I'm not saying it isn't a problem at all. I think I explicitly acknowledged that there is a problem where I said "the majority of the downside." But it is a problem that the free market can resolve. The free market can't resolve the doctor problem because the government literally requires the worthless undergrad degree to allow someone to practice medicine or else they'll be jailed.

[+][comment deleted]7mo20

The fine print

Universities get most of their money from students, so they will have strong incentives to cheat. As such the law forcing this change must be written in such a way as to minimise the viability of this strategy.

I've written an initial draft of some rules that attempt to tackle this. There's no need to wade through this if you don't want to, but I would appreciate it that if you intend to comment with a reason why this wouldn't work, you skim through this to see if it's already addressed.

Regulations governing granting of credentials.^[1]

Definitions and Student Acceptance:
a. Internal Student: An individual to whom the institution provides tutoring, instruction, or other educational resources intended to facilitate preparation for the credential’s assessments, irrespective of the student’s physical location or mode of study.
b. External Student: An individual seeking to obtain a credential solely by completing the required assessments, without receiving tutoring, instruction, or other educational resources from the institution specifically for said credential, beyond publicly available assessment information.
c. Selection and Acceptance: Institutions may establish their own criteria for selecting Internal Students, subject to applicable laws and regulations. Institutions may not deny access to assessments for any External Student who meets the basic registration requirements, except as specified in Section 17.a concerning resource limitations.
d. Assessment-Only Provider: An institution that, for one or more specific credentials, facilitates credential attainment exclusively through assessments as defined herein, without providing the tutoring, instruction, or other educational resources characteristic of an Internal Student relationship for said credential(s).
Application to Assessment-Only Providers: These regulations shall apply mutatis mutandis (with the necessary changes) to Assessment-Only Providers. However, specific provisions predicated on the existence of Internal Students for a given credential, or requiring direct comparison between Internal and External Student conditions, shall be considered inapplicable to that provider for that credential. This includes, but is not limited to, aspects of Section 7 (regarding equitable treatment relative to internal students), Section 10 (Cost Cap relative to internal tuition), and Section 18 (requiring outcome reporting disaggregated by internal/external status).
Accessibility of Credentials: Institutions offering credentials under this framework shall make such credentials accessible to any individual who successfully completes the required assessments, subject to the definitions and acceptance rules in Section 1 and the applicability rules in Section 2.
Confidentiality of Student Status: Institutions operating under this framework are prohibited from disclosing or confirming to third parties whether an individual obtained a credential as an Internal Student or an External Student.
Voluntary Compliance and Consequences of Non-Participation: Participation in this regulatory framework is voluntary. However, any institution electing not to adhere to these regulations in their entirety shall be prohibited from responding to any inquiry concerning, or otherwise confirming, an individual’s attendance at the institution or the conferral of any credential purportedly issued by said institution.^[2]
Scope of Assessments: Assessments may encompass, but are not limited to, written examinations, coursework, practical evaluations, and in-person examinations.
Equitable Assessment and Grading:
a. The content and grading standards of assessments shall be identical for Internal Students and External Students.
b. Institutions shall implement blind grading procedures wherever feasible to mitigate bias, particularly where both Internal and External students undertake the same assessment.
c. Grading timelines shall be equivalent for Internal Students and External Students; preferential expedited grading for Internal Students is prohibited.
Assessment Fees:
a. Assessments shall be offered at a price not exceeding the calculated cost plus a reasonable profit margin.^[3]
b. Cost calculations shall be based on verifiable methodologies and must be made publicly accessible.
Cost Minimization and Accessibility:
a. Institutions shall undertake reasonable measures to minimize their operational costs related to assessments and the ancillary costs incurred by students.
b. To mitigate inflated invigilation charges and reduce student travel burdens, written examinations shall, where practicable, be conducted at publicly administered examination centers adhering to standardized fee structures.
Cap on Total Assessment Costs: The total cumulative cost for an External Student to obtain a credential solely through assessments shall not exceed twenty-five percent (25%) of the standard tuition fees charged to an Internal Student enrolled in the full corresponding course of study.^[4]
Assessment Scheduling:
a. Institutions shall structure assessment schedules such that it is feasible for a student to complete all assessments required for a specific credential within a single academic year, defined as commencing August 1st.^[5]
b. Should unavoidable scheduling conflicts preclude completion within the designated timeframe, the institution must provide alternative assessment opportunities. The institution may levy an additional fee for such arrangements, not exceeding double the standard fee for the affected assessment(s).
Assessment Time Commitment:
a. The cumulative time mandated for participation in invigilated or time-bound assessments shall not exceed four (4) weeks per standard academic year attributed to the credential’s typical duration for Internal Students, calculated based on an eight (8) hour day, four (4) days per week.[6]^[6]
b. This limitation expressly excludes assessments, such as coursework, completed at the student’s own pace and timing.
Mandatory Public Disclosure via Government Portal: Institutions adhering to these regulations shall freely provide and maintain the following information on a designated, government-hosted website:
a. A comprehensive list of all credentials offered by the institution under this framework.
b. Precise and exhaustive requirements for the conferral of each credential.
c. For every assessment contributing to a credential:
i. A detailed description of the assessment format and a high-level overview of the grading methodology.
ii. A representative example of the assessment instrument and a corresponding example of a concise response meeting the highest standard (e.g., sample examination paper with model answers, video/transcript of an oral examination).
iii. A detailed syllabus outlining all topics covered, including the required depth of knowledge (e.g., specifying “group theory: basic definitions and results, Sylow Theorems, Orbit-Stabilizer Theorem” rather than merely “group theory”). It cannot cover material that is both not taught internally (where applicable) and will not be assessed.
iv. A list of all publicly accessible learning resources utilized by Internal Students (where applicable) (e.g., textbooks).
v. Where assessments cover material not readily available through public channels, direct links to, or provision of, said material presented in a manner sufficient for a capable student to comprehend with reasonable effort.
vi. All materials concerning assessments provided to Internal Students (where applicable) (e.g., past papers, marking rubrics, explicit guidance on examination content - including verbal guidance).
Information and Registration Availability Deadline: All information pertaining to a specific assessment, as detailed in Section 13.c, and the functional capability for students to register (sign up) for that assessment, must be publicly available via the designated government portal no later than one (1) calendar year prior to the scheduled date of the assessment.
Online Assessment Management Tools: The aforementioned government-hosted website shall incorporate functionalities allowing students to register for assessments (subject to the deadline in Section 14), receive notification of results, and track their progress towards credential attainment.
Notification of Changes: Institutions shall provide a minimum notice period of one (1) full academic year prior to implementing significant modifications to credential requirements or assessment structures, except in documented circumstances of exigent need.
Exceptions for Resource-Intensive Assessments:
a. For assessments necessitating scarce or high-cost resources (e.g., advanced laboratory facilities, human cadavers), institutions may, for an initial period of ten (10) years, limit the enrollment of External Students (where applicable) to ten percent (10%) above the capacity allocated for Internal Students (where applicable). This permissible limit shall increase by five percent (5%) annually thereafter. Acceptance of External Students under this limitation shall be granted on a first-come, first-served basis according to registration time. For Assessment-Only Providers, limits may be based on absolute resource capacity, subject to justification and oversight.
b. Institutions operating under this provision are mandated to actively develop and implement strategies to alleviate these resource constraints within the specified ten-year timeframe (e.g. facility expansion, year-round scheduling, assessment modality modification).
Reporting of Assessment Outcomes: All institutions subject to these regulations shall publicly report the precise distribution of marks and grades achieved in assessments. Where both Internal and External students exist for a credential, this reporting shall be disaggregated for these groups.
Appeals Process:
a. All students undertaking assessments under this framework possess the right to appeal assessment results to a designated governmental oversight body.
b. Said body retains discretion regarding the investigation of individual appeals but shall initiate a formal investigation upon observing an anomalous volume of appeals or statistically suspect distributions of marks/grades pertaining to an institution or specific assessment.
c. Institutions subject to investigation must fully cooperate and provide all requested materials related to the assessment(s) in question (including, but not limited to, examination papers, marking schemes, grading records). Failure to comply may constitute grounds for inferring malpractice.

Impacts

The impact that I'm most interested in, is that smart, driven kids are likely to realise they can save significant time and money by taking a degree externally instead of internally. I could even imagine it becoming a thing among the current high performers who all aim to get perfect GPAs, to instead finish their degree by the time they finish school.

(Similarly it removes the incentives that lead to the current rat race where teenagers need to rack up as many achievements as possible by the time they're 18 so that they'll get accepted to Harvard or MIT.)

When employers realise that's happening, an external degree is likely to be seen as a positive sign, implying that the candidate is capable and driven enough to pass the exams without requiring tutoring.^[7]

The short term impact on universities will be varied:

There'll be a lot of interest in credentials from the most elite universities. Desperate to protect their shine they'll both try to tighten up their assessment standards (good), and rule hack their way into making it as unfair as possible for external students (bad). Expect a few lawsuits before this settles somewhere stable.

Nobody is much interested in external degrees from the hundreds of average universities, so except for a handful of applicants each year they pretty much carry on as normal. Compliance is a burden, but they don't put in too much effort and the number of external applicants is low enough that they don't get sued over it.

A small number of average universities realise there's low hanging fruit here and gain a name for themselves as providing fair, achievable-but-rigorous assessments at reasonable prices. Most "normies" who want an external degree go for one of these. They also likely charge extra for online courses and material students can study in their own time. These universities cover a wide range of standards and some even try to challenge the elite universities, with limited success.

At the bottom end some universities gain a name for themselves as places that will give you a degree in whatever you want if you pay them and can spell your own name. Recruiters know to ignore them for the most part, but they allow you to tick the box of "have a degree" in some cases (or until the university gets added to the blacklist).

Meanwhile schools will pop up which teach students how to pass exams from other institutions, especially the prestigious universities and those that target large numbers of external students. These will attempt to compete on cost, speed, pass rate and experience.^[8]

Over the long term average universities will realise that they are spending a lot of effort complying with these regulation and setting exams, while there's institutions that have a name for themselves as providing high quality credentials specifically geared towards external students. They will consider teaching their students the curriculum for these institutions instead of setting their own exams. Over time, I expect that we'd have an increasing separation between institutions that provide credentials but no teaching, and those that teach but do not provide credentials, much like A levels and GCSEs in the UK today^[9]. This helps solve the problem where recruiters don't know which degrees are worth anything, since the market will consolidate around a small number of institutions who have a reputation to maintain.

Concerns

This entire proposal is predicated on the assumption that the most important thing that universities provide to employers is the ability to confirm you are clever, driven, and have relevant skills, and that exams are a sufficient way to test that.

I think this is probably true for software engineers, and consultants. However it's arguably false for medical professionals - a nurse who's read every textbook and can ace every exam is likely to be useless if they've never drawn blood before . Of course we could test that in an exam, but these regulations actually push away from such tests since they are harder to scale (you need to find suitable victims who are willing to have blood drawn in an exam).

I'm not sure where I stand on this, but as ever I think "Do not let perfect be the enemy of good". Making these changes will be hugely valuable even if we carve out doctors and a few other professions.

^{^}

These were all written by me, then reworded by an LLM, and checked over again by me.

^{^}

This means a swimming school could continue to grant certificates to students who successfully swim 100 metres, because they do not much care if somebody falsely claims to have such a certificate from them. For MIT, someone falsely claiming they have a BSc is a much more serious problem.

^{^}

To prevent institutions splitting up a credential into hundreds of tiny exams and thus gain greater profits, a reasonable way to limit the profit margin is to say that total profit for all assessments take by an public assessee for a credential which would usually take X years when studied in house must not exceed X thousand dollars (rounded up).

^{^}

This limits total monkey patching institutions can do to drive up costs.

^{^}

This is to prevent institutions deliberately arranging exams so assessees are forced to spend the full 3 years gaining the credential, reducing the value of not studying in house.

^{^}

This is to prevent institutions making an exam that lasts one year but includes lectures, or something similar.

^{^}

If I turn out to be wrong and recruiters still favour internal degrees for whatever reason, we could make the type of credential (internal vs external) a protected characteristic they can't ask about (although they could still probably guess by asking questions like what were you doing last year).

^{^}

If we follow the same principles as in regulation 5, they can choose to select who they teach, but cannot answer queries about whether anyone was taught there.

^{^}

In the UK AQA, OCR and Edexcel all set their own curriculum and exams for GCSE/A-level subjects, and schools choose which ones they want to use for each subject.

LESSWRONG
LW

LESSWRONG
LW

13

How to end credentialism

13

13

The fine print

Impacts

Concerns