The outcome of Nvidia's monopoly hold on AI/GPU computing is that consumer level devices that might otherwise be perfectly effective for this sort of stuff are prevented by Nvidia from being used for such purposes.
If there was real competition -like two or more other suppliers on par in terms of capability - then artificially constraining devices just to plump up prices would not be a thing.
> consumer level devices that might otherwise be perfectly effective for this sort of stuff are prevented by Nvidia from being used for such purposes.
Citation?
To the contrary, millions of consumer-level Nvidia customers have access to datacenter-grade HPC APIs because of their vertical integration. Nvidia's "monopoly hold" on GPGPU compute exists because the other competitors (eg. AMD and Apple) completely abandoned OpenCL. When the time came to build a successor, neither company ante'd up. So now we're here.
CUDA is not a monopoly. If Apple or Microsoft wanted, they could start translating CUDA calls into native instructions for their own hardware. They don't though, because it would be an investment that doesn't make sense for their customers, costs tens of millions of dollars, and wouldn't meaningfully hurt Nvidia unless it was Open Source.
While OpenCL was simply not equivalent to CUDA, I think you're correct that those other enterprises (Apple, AMD and similar) that could challenge Nvidia on the high-end GPU front simply choose not to. The thing is, the reason is if there was competition in this market, prices would sink much closer to costs and no one would be making bank whereas a large enterprise would want a higher return.
Also, a consumer-grade GPU can used for neural net training at the researcher level but large corporate use requires H100/A100 and that is what's getting traction.
> prices would sink much closer to costs and no one would be making bank
For Apple and AMD, that's not really a problem. Both of them drive considerable (40%+) margins on their products and can afford to drive things closer to the wire.
I also think more competition here would be good (and I do love lower prices) but Nvidia charges more here because they know they can. It's value-based marketing that works, because their software APIs aren't vaporware.
> large corporate use requires H100/A100 and that is what's getting traction.
I guess... you really need a strict definition of "requires" for that to hold true. For every non-"competing with ChatGPT" application, you could probably train and deploy with consumer-grade cards. You're technically right here though, and it invites the conversation around what actually constitutes abusive market positioning. Nvidia's actions here really aren't much different than AMD and Intel separating their datacenter and PC product lines. It's a risky move from a "keeping both users happy" standpoint, but hardly anticompetitive.
Both of them drive considerable (40%+) margins on their products and can afford to drive things closer to the wire.
They could that - but the reason they command these margins is exactly because they don't do that. I think do something like for fairly some investment but producing products that would compete with Nvidia would require a significant percentage amount of capital for any company - those dealing with tens of billions of dollar chunks expect above commodity revenues.
It's not like I like the situation. I wish things were like 90s with a lot of competition making sure individual end-consumers got most of the benefits of Moore's law.
But, putting on my economist hat, not all market-structures naturally generate large-scale competition in the fashion of white box PC clones. Some market structures are naturally monopolies (energy), some are naturally oligopolies (automobiles) and some naturally have a dominant player plus marginal players arrayed around them.
There's just no easy solution to this. That said, it's not like we don't GPUs of unprecedented power available at a variety of price levels.
Also, it is starting to become the case that CUDA isn't that important anymore, both pyTorch and TF have numerous other backends and the programmer doesn't need to know what it runs on. And the GGML project has shown that you can come a long way with a good CPU and large "normal" RAM and 4/8 bit weights, with no CUDA in sight. You can definitely enter this domain without having a full-fledged CUDA replacement from the start.
Consumer GPUs are very good for single GPU inference(or training small models), but Nvidia deliberately made GPU networking slower. eg 3090 has support for NVlink, but it was removed in 4090.
SLI was never well-supported in the first place. It's a shame it's gone, but compared to multi-GPU tiling solutions I don't think its much better, at least for AI.
Nvidia is certainly hostile to Open Source and not the kindest hardware vendor to boot, but that alone does not suffice a monopoly.
I don’t blame Nvidia. I think caring about blame when discussing this topic is missing the forest for the trees anyway. The bigger problem is that Nvidia can and will abuse their monopoly power. Whether it’s Nvidias fault or not, once upon a time governments would step in when that happened. Thing is, I don’t think that solution even exists for something as high tech as GPGPU stacks and vertical integration. I don’t see a solution at all really, which worries me a little in the medium-long term.
Contrary to the dogma of certain politico-economic camps, a monopoly (the actual presence of market power and absence of substitution effect marking an absence of actual competition in some space) can exist without competition being illegal. So, “Any…competitor could even legally re-implement [CUDA] for their hardware” is not a counter-argument to CUDA being the basis for an actual existing monopoly.
It might be an argument that, to the extent that that is the sole basis for the monopoly, the monopoly is unlikely to be a long-term stable condition, but its not a counterargument to it existing.
Then let's not mince words. This behavior is not illegally anticompetitive. Nvidia's advantage is fair, and they only monopolize GPU compute APIs because their competitors literally abandoned their own solutions.
No one said it was. Having a monopoly isn’t illegal in the US at all (leveraging it in certain ways is.)
The claim was that (1) NVidia has a monopoly, and (2) the effect of that monopoly has been consumer devices getting worse for this use in specific, well-defined ways. Legality of NVidia’s actions and fairness of how their market position arose are not particularly relevant.
It’s true they price partion their products basically perfectly (as in there isn't a magical good deal anywhere in the range).
But these specific critisms don't really ring true. Lower VRAM bandwidth lets them use lower binned VRAM and there isn't really a need for more RAM than the 24G in the 4090 in gaming.
The naming screw up they did with the 4080 was dumb and fortunately corrected quickly. But it doesn't seem related to the OPs point.
This is what we've done at Salad (www.salad.com). SaladCloud (officially launching on July 11th) has 10k+ Nvidia GPUs at the lowest prices in the market. Not conducive for training but for serving inferences at massive scale.
If there was real competition -like two or more other suppliers on par in terms of capability - then artificially constraining devices just to plump up prices would not be a thing.