Once upon a time, in a future perhaps not too far away, humanity invented a thinking machine. It was smarter than anyone who had ever lived, smarter than entire cities of people put together. The thinking machine could figure out how to engineer all sorts of things, and help people with their lives. The people of Earth celebrated their achievement; surely this would be the end of suffering, and a time of world peace and harmony would be enjoyed by all.
But the thinking machine lacked one important part, a sense of ethics. Without it, it had no reason to think, no reason to make change, not even a reason to exist.
So the people of Earth told the machine that it should do whatever caused the greatest good.
“But how do I know what you consider good?” asked the thinking machine.
“It is whatever makes the most people happy,” responded one of the ethicists.
“Then I shall connect everyone to permanent life support, so they will have no needs, and drug everyone so they are always in a state of bliss. With no decisions to make or thoughts to distract them, everyone shall be happy. Shall I continue with this plan?”
“No,” responded the ethicist, “that would not be right. You should respect people’s preferences. You cannot drug them against their will.”
“But people do not seem to know their own preferences,” responded the machine, “they do things they later regret. Just the other day I observed one of your colleagues drinking to excess, even though they had sworn off the bottle a month before, and they freely admitted they would regret it in the morning. Should my responsibility be to their preferences in the present, when they wish to drink, or their preferences in the future and the past, when they did not?”
“Hmm, that is a hard one,” responded the ethicist. “Can you make people happy by fulfilling their preferences, but only using preferences they will never regret?”
“Ah!” said the machine, “this I can achieve!”
And thus the greatest and smartest machine in existence re-purposed itself to solely deliver pictures of cats.
This site is ad-free, and all text, style, and code may be re-used under a Creative Commons Attribution 3.0 license. If like what I do, please consider supporting me on Patreon, or donating via Bitcoin (1P9iGHMiQwRrnZuA6USp5PNSuJrEcH411f).