Have you ever looked at the code produced by your favorite LLM?
And above all its performance, between the verbosity of the generated code, its production cost and the execution speed of the script?
I tested, via OpenCode Go (10 USD/month), several LLMs on a simple prompt: “Make a PHP script that consumes a 100,000-entry XML file as quickly as possible.”
In the end, I tested 6 LLMs (I currently use DeepSeek v4.1 Flash heavily, which I find extraordinary for coding).
DeepSeek v4.1 Flash, Grok 4.7, Kimi K3.1 Pro, GPT-6 Luna, Gemini-5 Argon and Claude Haiku 5.5.
And the results on this request are as follows:
The most compact code is generated by GPT-6 Luna, in 29 seconds, for a cost of 0.00156 USD! This is my big surprise of the test: OpenAI is 10 times cheaper with Luna than DeepSeek…
The fastest script is produced by DeepSeek v4.1 Flash, which reaches 528,000 entries/s on file processing.
It indeed takes only 189 ms to parse the 100,000 XML entries!
It chooses to use XMLReader in streaming… and is the fastest.
Claude Haiku 5.5 is also inexpensive in this benchmark, generates the code very quickly (17 seconds), versus 115 seconds (almost 7 times more) for DeepSeek, but produces code that is among the slowest to execute.
In terms of performance / speed / cost ratio, DeepSeek is ahead on this one, if one prioritizes the execution speed of the produced code.
Grok is very expensive (20 times more than DeepSeek), but comes second in performance.
Luna is 10 times cheaper than DeepSeek, to come fourth in performance, with extremely fast production (29 seconds).
What is interesting in this little exercise is that we are not simply measuring an LLM’s ability to produce functional code, but also its ability to make the right technical choices, to avoid unnecessary verbosity and to generate a truly performant program.
And, clearly, the most expensive model is not necessarily the one that produces the best code.
This also illustrates another problem: which models for which tasks…
(Good) orchestration is going to become an increasingly important issue.
LinkedIn Sources
- Profile: Pierre Jean Duvivier dit Sage (LinkedIn)
The post Have you ever looked at the code produced by your favorite LLM? appeared first on les plumes qui mordent.
Frequently Asked Questions
FAQs Looking at the Code Your Favorite LLM Produces
1 What does it mean to look at the code an LLM produces
It means actually reading running and reviewing the code an AI assistant gives you instead of copying and pasting it blindly
2 Why should I bother reading AIgenerated code
Because LLMs can be confidently wrong Reading the code helps you catch bugs security issues and logic errors before they reach your project
3 Do I need to be an expert to review AI code
No Even basic reading helps you can spot obvious problems like hardcoded passwords missing error handling or code that doesnt match what you asked for
4 What kinds of mistakes do LLMs commonly make in code
Typical issues include hallucinated functions or libraries outdated APIs offbyone errors missing edge cases and insecure defaults
5 Can AIgenerated code contain security vulnerabilities
Yes Models sometimes produce code with SQL injection risks weak input validation or exposed secrets Always review before shipping
6 Why does the code sometimes reference functions or packages that dont exist
LLMs predict plausiblelooking text so they can invent APIs This is called hallucination always verify that imports and functions actually exist
7 Is AIgenerated code usually correct
Its often close but usually close isnt good enough for production Treat it as a first draft not a final answer
8 Whats the fastest way to check AI code
Run it on a small test case read it line by line and check that every import and function call is real Then test edge cases
9 Should I test AIgenerated code differently than my own code
Not really but test it thoroughly especially edge cases and error paths since the model may not have considered them
10 Can I trust AI code if it looks clean and wellformatted
No Clean formatting says nothing about correctness Some of the most convincinglooking AI code is subtly broken
11 What are the benefits of reviewing AI code instead of rejecting it outright
You get the speed of AI generation plus the safety of human judgment and you often learn new patterns or approaches along the way
12 How do I know if the AI used an outdated library or API
Check the
