Much of the work done is fully open source and are liberally licensed.
DeepMind and OpenAI have a bad rep in this regard.
But a lot is available for free (as in beer and speech).
And most of the research papers are released in arXiv. It's very refreshing.
The bottleneck is not the knowledge or code, but the compute. People are fighting this in innovative ways.
I have been an inactive part of Neuropark that first demoed collaborative training. A bunch of folks (some of them close to laypeople) ran their free Colab instances and trained a huge model. You can even utilize a swarm of GT1030s or something like that.
Also, if you have shown signs of success, you are very likely to have people willing to sponsor your compute needs, case in point- Eluether AI.
The situation is far from ideal with this megacorps rat race [0], and NLP research being more and more inaccessible, but it is not completely dark.
[0]: I, along with many respected figures tend to think that this scaling up stuff approach is not even useful. We can write good prose with GPT-3 nowadays, that are, for all intents and purposes, indistinguishable from text written by humans. But we are far, far away from true understanding. These models don't really understand anything and are not even "AI", so to speak.
The Transformer architecture, the backbone of all these approaches- is too brute-force-y for my taste to be considered something that can mimic or, further- be intelligent.
DeepMind and OpenAI have a bad rep in this regard.
But a lot is available for free (as in beer and speech).
And most of the research papers are released in arXiv. It's very refreshing.
The bottleneck is not the knowledge or code, but the compute. People are fighting this in innovative ways.
I have been an inactive part of Neuropark that first demoed collaborative training. A bunch of folks (some of them close to laypeople) ran their free Colab instances and trained a huge model. You can even utilize a swarm of GT1030s or something like that.
Also, if you have shown signs of success, you are very likely to have people willing to sponsor your compute needs, case in point- Eluether AI.
The situation is far from ideal with this megacorps rat race [0], and NLP research being more and more inaccessible, but it is not completely dark.
[0]: I, along with many respected figures tend to think that this scaling up stuff approach is not even useful. We can write good prose with GPT-3 nowadays, that are, for all intents and purposes, indistinguishable from text written by humans. But we are far, far away from true understanding. These models don't really understand anything and are not even "AI", so to speak.
The Transformer architecture, the backbone of all these approaches- is too brute-force-y for my taste to be considered something that can mimic or, further- be intelligent.