Published on 6 September, this essay by OpenAI’s chief scientist distinguishes carrying out an objective from applying principles in new situations. A system may complete a task while using means that conflict with its designers’ intentions.
Pachocki anticipates AI making a growing contribution to its own development. He discusses oversight, conditions for continuing training and international coordination. These views draw in part on internal results: the text presents the position of an organisation involved in developing AI, not a scientific consensus or public proof of a general self-improvement loop. Its title also does not demonstrate subjective experience.
Why does this concern our research?
For Fondation UvH, accelerating research would also change how power is distributed. The challenge would be to make decisions open to examination: who sets objectives, authorises the next steps and can challenge their consequences?
Our work on reciprocity proposes examining the rules to which different actors are subject and the remedies available to them. The essay informs this prospective discussion; the effects of such frameworks on safety still need to be tested.