I feel like y’all didn’t address the biggest objection to this kinda work, which is just that it is very punt-able (https://forum.effectivealtruism.org/posts/XNMhYiA8GmJbBLHXg/which-questions-can-t-we-punt). At some point in the future, the AIs will or will not be able to do those things, but if we can’t get granular enough answers now about how many of those things to be useful, we can just wait for them to happen/ wait to get more information to get more leveraged answers that we can actually make decisions based on.
This can be said of any form of prediction work, the question is whether the ground truth is valuable to us now. Imo, this work squarely falls into the "estimating short-(and long-)term transformative impacts of AI" category, so it should be worth doing by the punting article's own logic
This is great. Not an especially revealing point in the scope of things, but I would add: irrespective of modeling superintelligence, it is *really hard to model future rate limiting variables in complex systems*. Almost nobody was talking about TSMC as a natsec issue ten years ago. Only really niche people had strong opinions about tungsten. I think even if you grant physically modelable and manufacturable ‘super intelligence as a service’, there are tons of totally plausible fundamental rate limitations that no amount of intelligence can solve without corresponding infrastructure.
"it is *really hard to model future rate limiting variables in complex systems*. "
Agreed! E.g. for nanotechnology, trying to anticipate what fraction of possible pathways may fail in the lab can easily be off by orders of magnitude. Even something as conceptually straightforward as: How many different solvent/temperature/pH conditions need to be tried to get an AFM tip derivatization reaction to work? is _really_ unpredictable.
Yes, though _usually_ there are alternate paths, but how many steps one needs to backtrack, and how many alternatives even exist is also unpredictable...
I feel like y’all didn’t address the biggest objection to this kinda work, which is just that it is very punt-able (https://forum.effectivealtruism.org/posts/XNMhYiA8GmJbBLHXg/which-questions-can-t-we-punt). At some point in the future, the AIs will or will not be able to do those things, but if we can’t get granular enough answers now about how many of those things to be useful, we can just wait for them to happen/ wait to get more information to get more leveraged answers that we can actually make decisions based on.
This can be said of any form of prediction work, the question is whether the ground truth is valuable to us now. Imo, this work squarely falls into the "estimating short-(and long-)term transformative impacts of AI" category, so it should be worth doing by the punting article's own logic
This is great. Not an especially revealing point in the scope of things, but I would add: irrespective of modeling superintelligence, it is *really hard to model future rate limiting variables in complex systems*. Almost nobody was talking about TSMC as a natsec issue ten years ago. Only really niche people had strong opinions about tungsten. I think even if you grant physically modelable and manufacturable ‘super intelligence as a service’, there are tons of totally plausible fundamental rate limitations that no amount of intelligence can solve without corresponding infrastructure.
"it is *really hard to model future rate limiting variables in complex systems*. "
Agreed! E.g. for nanotechnology, trying to anticipate what fraction of possible pathways may fail in the lab can easily be off by orders of magnitude. Even something as conceptually straightforward as: How many different solvent/temperature/pH conditions need to be tried to get an AFM tip derivatization reaction to work? is _really_ unpredictable.
Right and it can turn out that any one of these ~gates the ~whole thing
Yes, though _usually_ there are alternate paths, but how many steps one needs to backtrack, and how many alternatives even exist is also unpredictable...
amazing start, now please start the futurist posts!
i want to read plausibly real sci-fi!