You know, discussions like these is why I mostly keep of MacRumours these days. I really don't want these boards to adopt a similar culture.
In my opinion, it is very important to understand the purpose behind the questions. For instance, why are you posing this specific question and not a differently phrased one? What is that that you care about? The fact that the laptop does not throttle on battery, or the fact that a laptop is useful on battery? It is always possible to manipulate the question so that only one answer is possible. Are we achieving anything constructive with it? Hardly...
Okay, let me start by asking similar of you. What is the purpose behind your sentence regarding “discussions like these is why I keep off MacRumors…” Are you trying to insinuate something? If you are, you should directly say it. You are an older adult presumably, and I am not going to spend my time speaking for you. If you are trying to say something about me or what I write, say it directly.
Second, my purpose is clear. I made my comments in the context that I have watched this forum talk about M3 and pontificate about “its gonna have this or that,” and then when it came out and it did not initially meet this forum’s expectations, there was a collective shitting on by this forum. To the extent of MacRumors style? No, but a lot of this forum set up their own weird expectations of this chip and then got all in a tizzy because it wasn’t. I don’t want to argue about ”no we didn’t.” You can go look at the original M3 posts months before, starting with Cliff Maier speculating it’s going to have X amount of performance increase. I love reading what Cliff writes, but that was foolish to do. Sorry. It just was.
Furthermore, I have been reading stuff that a lot of people here have written be it on macrumors or here since M1, and the tone the last month or so has been markedly different beyond normal deviation.Again, it felt like it was more caught up in stupid narratives pushed by the media in addition to it Initially not living up to whatever people thought it was going to be. Those concerns never really bore out, for the record.
That Apple silicon can X with Y given watts with Z amount of battery life is the entire point of these discussions. That is all there is to discuss chips: what they can do, how much power it takes, and what’s the practical battery life and portability etc. All discussions fit within these three broad categories. There are other discussions, like X86 vs.ARM, which Cliff, for example, has been very nice in providing expertise in explaining.
With that being said, this situation is why I said what I said. You kept going on saying that Apple’s M chips are X or Y, and I directly went against that because I thought it was kind of strange to say that without the broader context of what Apple is offering relative to the market and products sold today. I’m not saying Apple is perfect nor better than Nvidia at everything. I have very specifically outlined what I believe Apple is offering, and that it is not “behind” And gave the reasons for that. 100 seconds vs 8 seconds is a huge difference and would be horrible without context. The context being that to achieve 8 seconds, you need a 3 fan, plugged in permanently, 400 Watt GPU that costs $1,600 at least. I thought I made a very good comparison and illustration to make the point I was trying to make clear, but I guess not.
This thread has given multiple benchmarks and examples of what Apple silicon is capable of. I added that it’s in the context of a thin, light notebook that can do this stuff on the go. I am not going to speak for you; you can do that yourself. I am just saying there was too much criticism of it (Again, refer back to my second paragraph) without the context of everything. I wanted to provide input, so I did and I thought I contributed something valuable. You can have your own opinions about what I write I guess.
Just because I speak out agains maximalism doesn't mean that there is a "sharp turn in my commentary". Apple did some impressive work on the GPU front with M3, and we can confidently claim that they overtook Nvidia in some key areas. But this doesn't change the fact that their GPU lack raw GEMM power and that the memory bandwidth could be better.
I addressed this above. I have no clue what maximalism is, but I am going to presume you think I was making Apple silicon look better than it is? All I did was use the benchmarks this thread wrote, and added that it can be done in a notebook unplugged. That level of performance is not something you can get in a windows laptop unplugged, and the only thing I was slightly dramatic with was deleting my account. But I actually meant that. I hate Macrumors and it’s not like I have an allegiance to you guys or this website. My point I was making with deleting my account etc is to show me a windows laptop capable of the same thing on battery for more than a minute, and I won’t speak about it anymore. I have yet to see windows offer this sort of thing.
Apple said awhile ago notebooks are their primary mac that customers buy. That Apple needs to increase their speed by more than 10 X to catch up to a 4090 in a particular test is useful to know but not entirely, because again, what M3 offers is amazing for a notebook. comparing it to a full throttle unlimited power source with gigantic ass fans and it costs thousands to a notebook is useful to know what it’s capable of, but of course is ridiculous and unfair to compare beyond saying this is what Apple could improve To get to this level. To compare the two and insinuate through the denial of what ive writtten that Apple’s is incapable is ridiculous. The tone of what has been written is what’s different, and recently a bit of the content as well.
What Apple silicon notebooks offer is incredible given the context, and that’s only ever been the point I’m trying to make.
Regarding GEMM, I’ll admit I didn’t read every comment in this thread, but I don’t recall this even being a point made before you replied to mine. Again, while I appreciate the analysis, the tone comes across differently with M3. Maybe I just ignored your posts in the past and they’ve always come across as too critical. I actually think instead that it’s just you’re more critical this gen, and it seems like there was way too much speculation and hope for X Y Z on M3. Apple shipped what it shipped. I found it awfully surprising this forum thought M3 was a disappointment when it launched (and I say that in the context of not only speculation and comments regarding M3 before it launched, but also the tone and commentary around M1 and 2) given that was stupid media narrative and not focusing on the actual chip. Then the chip came out, and I predicted to myself that people would then again realize it’s good, and then of course the commentary toned down as expected. And it did, mostly. I still get the hint of some being disappointed and I don’t understand it
personally.
Absolutely, you won't get any argument from me here. Large amount of RAM with uniform performance characteristics is undoubtedly a unique advantage of Apple Silicon and will be an important asset going forward. And I also fully agree that the platform is a good foundation for future improvements. They just need more bandwidth and more GEMM compute.
Great, and I am glad and not surprised you agree. That being said, I did not claim ever that Apple did not need to make improvements. Everyone does. Apple has and will continue to relentlessly iterate and improve and I am excited to see what’s next. I mentioned Apple’s chip is technically 10X slower in that one test. Objectively it’s slower. In the context of the situation regarding windows notebooks, and also that you needed a 400 W 3 fan system plugged in permanently, it’s powerful and impressive for an on-the-go notebook, again, my main point I was trying to make that I didn’t feel was being articulated nor recognized. Regarding GEMM, see what I wrote above as well
This is a type of video that makes me throw my hands up in frustration. This is not information, it's content. One has to sit through 10 min of a person talking to get something that could be summarised in a few sentences. And having skimmed though the video, I still don't know what these results mean. There is a standard way to report performance of LLMs: tokens/second. Yet here he uses some arbitrary time measure on some arbitrary task. This is simply not helpful. There is no analysis. Is the M3 Max limited by the bandwidth? Is it limited by compute? Why not load the software into the Xcode instruments and look at the GPU profiler stats?
Uh, sure. I hate a lot of YouTube at the moment and that longer videos aren’t usually necessary. That being said, it’s now coming across as completely dismissing the point. You didn’t even watch the video? Your time, your choice. But it demonstrated the point I have been trying to make this whole time, that Apple silicon lets you do stuff not possible on other notebooks. I did not say it was the most technically apt or something like Anandtech would produce.
That being said, PRACTICALLY speaking he demonstrated something that needn’t anything beyond what he did.He tried to do a task the same way on all 3 machines, and Apple silicon’s unique architecture let it beat out, PRACTICALLY speaking, the 4090. Your earlier comments against the rest of what people were saying in this forum was like you were discounting that there was no way M3 could compete with 4090. if that was not what you were saying nor implying, directly speak and articulate that. However, given what you wrote earlier I stand behind my inference. You can change your mind or clearly say what you think. By the way, I am not saying you need to read or watch anything I suggest. That being said, your critique came across as fallacious and unconvincing to me, personally. YouTube has turned to be generally stupid as fuck; however, there are almost zero videos doing what he did, and just because it could have been better doesn‘t mean it’s invalid.
The only thing we learn from the video is that the Max with 128GB of accessible RAM has less of a performance penalty compared to a GPU with a smaller RAM pool (duh!). We still have no idea what this means in practical terms though. Is the performance of the 70b model sufficient to do relevant work? How does this compare to CPU inference speed? Would it be cheaper/more productive to build a workstation desktop or to rent a cloud computer? I can imagine a bunch of real world scenarios where an MBP is the best tool for the job. I can imagine even more scenarios where it is not.
I am segmenting this out from my reply above to specifically address certain things that weren’t already addressed by my above paragraphs. First, I already addressed your really strange critique of local vs cloud. I am starting to believe that I correctly interpreted this situation beginning with my first paragraph in this post, and you’re unfairly critiquing the M3 in ways beyond saying ’oh here’s something about it, what it offers, not, etc.’ You straight up ignored what I wrote regarding local vs. cloud. And now you’re back To saying this weird local vs cloud. There are benefits to doing things locally. DUDE, no one — including me — is claiming you’re building a massive 1 trillion parameter language model with a single M3 notebook. You’re writing questions In rapid fire and succession, but none of them really make sense to what I’m writing nor trying to argue. They are valid questions on their own but not to what I wrote.
And we DO know what it means practically speaking. If I am loading a model onto my computer for purposes of my choosing, if I asked the model to do X task, I will want a MacBook with a lot of unified memory. M3, even on tasks in that video that do not exceed the memory capacity of 4090, M3 was not far behind and yet relatively sips power compared to a 4090. There are valid and good reasons to want local models on a computer. You seem so obsessed with doing massive things that no GPU, including said precious 4090, can do without thousands of them linked, that you have now twice completely ignored that I said this point on local vs cloud.
These things are not as simple as getting the larger number. One needs to look at them in the context an actual problem. Otherwise we are lost in pointless microbenchmarking and e-peen measuring.
I agree with this, but it makes no sense given this converstion. You reply and critique what I gave and said it needs more detailed and nuanced benchmarking, but then say we would be lost in pointless microbenchmarking if we do not look at it in the context of an actual problem. I have been violating my own personal writing style and literally writing in capital letters “PRACTICALLY” repeatedly when making my point. This is why one should watch something before critiquing it. He tried to do a simple task and the 4090 was not capable of it at a certain point because it lacks the features and architecture that M provides. That doesn’t mean Nvidia doesn’t have unique features or performance that do tasks faster or better than Apple at the moment.
Thanks, this is very kind of you! It wasn't all to bad to be honest, just taking a while to get back to 100%. Ghastly weather isn't helping either. Can't wait to get back to the gym.
I am presuming you are feeling a little better, and I am grateful for that. I hope it continues like that. I am wishing you well.
However, because I do give a shit about you (and others here) , I must risk the appearance of contentiousness vs. my care about you, and say again you ignored something I wrote.
I am violating my own personal rules to not address nor discuss the ongoing pandemic on the internet, but
I told you to not exert yourself. Let me say again and be more clear.
Do not go to the gym or exert yourself physically nor psychologically for at least 4 weeks after you test negative consecutively. If you do, do not be surprised and/or think what follows is ‘confusing’ or ‘mysterious.’ If you ever get infected again, do not exert yourself, even if you ignore me this time and end up luckily not suffering. You can still suffer long term without exertion following infection; exertion shortly thereafter makes it even more likely. I will not speak further regarding this subject matter on this forum nor the internet.
However, I am open to hearing continuous updates how you are doing though, and importantly, interested because I genuinely care. That also goes for anyone else on this forum.
I don't, because quality information is very hard to find. But we know GEMM performance on Apple GPUs and Nvidia GPUs and it's not even close (for now at least). To make it clear, I think a lot of the Tensor Core rhetorics is mindless flag waving, since the GPU is limited by the memory bandwidth for a large class of problems. Which is why model quantization and intelligent data selection are going to become increasingly important for practical ML as we go forward. I don't think that Apple needs to chase Nvidia's Tensor Core performance levels. But support for lower-precision data types, quantization, and sparsity will be very important. I'm sure they are cooking something like that.
I appreciate what you wrote and I have nothing else to say other than I liked reading this. it came across as informative and a little bit more objective vs. some of the recent posts.
Precisely. What makes Apple so formidable is their ability to plan and execute. Looking back at the timeline of their advances, it becomes clear that some things are planned many years ahead. And it makes it easier to see what is likely to come in the future. I believe I spent enough time looking at Apple GPUs to have an idea what's coming. There is a logical path there. Or maybe I am just seeing ghosts, also possible
This wasn’t in response to me, even though I said something similar to the person you replied to In one of my comments.
I agree. Apple is an extremely unique company, and I am grateful a team like them exists! Their ability to do stuff like this is unparalleled not only in a tech company but teams period. It deserves an entire article on here dedicated to that topic, but I have written a lot, and I am tired. That is why I have never likes when idiots, like Macrumors, on the Internet irrationally hate Apple and try to change how they operate and their values. It pisses me off.
When people let Apple be Apple, the world is better off.