Not so much happened this week in releases compared to last weeks Google I/O and Claude 4 series. This time I also have no topic of the week deep dive, because I suffered a back injury this week. But thanks to NotebookLM, which is becoming a mainstay in my learning, i managed to get a few of my best picks of the week together. I also try to add more non-AI picks in the Other topic in coming versions, because only being able to talk about AI can be quite boring. This time I added a video talking about the financial and demographic challenges of Japan and how this will affect the whole world because of (but not only) the Yen Carry Trade.
For me the most interesting releases this week are an updated DeepSeek-R1 (with a smaller distilled version for mortals) as well as the circuit-tracer package from Anthropic, which attempts to find interpretable patterns in LLMs. Also the full system prompt of Claude 4 Sonnet was leaked on Github, showing how “dynamic thinking” or “research” capabilities are enforced in the model. The RAG world is also jumping on the reasoning and agent train, which I include here with a short paragraph about a blog post from LLamaIndex about Agentic Retrieval. Another release that I found this week is an LLM, that was trained to understand images by a new paradigm called VoRA. VoRA replaces the usual external vision encoder of multimodal LLMs with internal vision-finetuned LoRA layers ( coined VoRA).
Besides that, the most discussed paper this week was the “Spurious Rewards” paper. The authors of the paper showed that a model can achieve high accuracies in math problems after post-training RL with meaningless rewards. Another interesting story this week was a report by Invariantlabs, in which they showed how you can inject malicious prompts into MCP workflows of users of the MCP Github Server to expose private information. Also have a look in the book Reshuffle, from which I discuss a chapter below.
I hope you learn something and wish you a great week!
Releases
DeepSeek-R1-0528 release [10]
DeepSeek-R1-0528, an updated version of DeepSeek-R1, was released silently on Huggingface. There is the huge 685B main model and a 8B distilled variant with Qwen3 as the base model. The main model is close benchmarks to the other main models like o3 and Gemini-2.5-Pro-0506. The model does not have a different architecture than the previous R1, the improvements came solely from post-training Reinforcement Learning. Interestingly, now the base model of choice (not only for DeepSeek) is not a LLama model but Qwen models.
Vision as LoRA [1]
Vision as LoRA (VoRA) is presented as a novel method for converting Large Language Models (LLMs) into Multimodal Large Language Models (MLLMs). Unlike traditional MLLMs that rely on external vision encoders to process images, VoRA integrates visual capabilities directly into the LLM itself. This is achieved by applying vision-specific Low-Rank Adaptation (LoRA) layers within the LLM's Transformer blocks. During training, only these new LoRA layers and a lightweight visual embedding layer are updated, while the original LLM parameters are frozen. A key technique used is block-wise distillation from a pre-trained Vision Transformer (ViT), which helps transfer visual knowledge and accelerate training by aligning intermediate visual representations between the LLM and the ViT. For processing images, VoRA replaces the LLM's standard causal attention mask with a bi-directional attention mask for vision tokens, which is found to improve performance by better capturing contextual relations within the image. This approach allows the trained LoRA layers to be seamlessly merged into the LLM during inference, resulting in near-zero additional computational cost or memory overhead. It also enables the model to handle arbitrary image resolutions, leveraging the LLM's natural ability to process variable-length sequences. While designed to match the performance of conventional encoder-based MLLMs with sufficient pre-training data, its performance on tasks requiring extensive world knowledge may be limited by data availability rather than architectural constraints.
Claude 4 Sonnet Full System Prompt [5][6]
Anthropic traditionally publishes the system prompts of their models in their release notes [5]. It is a good template to adjust your own system prompts for Claude models that you use and also to see what the previous models did wrong. Interestingly, it entails a passage for giving good prompting advice and recommends few-shot examples.
being clear and detailed, using positive and negative examples, encouraging step-by-step reasoning, requesting specific XML tags, and specifying desired length or format. It tries to give concrete examples where possible. [5]
I did not see in my personal use cases that using positive and negative few-shot examples in prompts increased the overall accuracy.
They also made explicitly sure that Claude does not follow the Sycophancy road that ChatGPT took a few weeks ago
"Claude never starts its response by saying a question or idea or observation was good, great, fascinating, profound, excellent, or any other positive adjective. It skips the flattery and responds directly." [5]
The tools part is missing in the official release notes but the full system prompt was apparently leaked on Github [6]. Anthropic also has developed an internal Markdown standard called <antml>
<antml:thinking_mode>interleaved</antml:thinking_mode><antml:max_thinking_length>16000</antml:max_thinking_length>
If the thinking_mode is interleaved or auto, then after function results you should strongly consider outputting a thinking block. Here is an example:
<antml:function_calls>
...
</antml:function_calls>
<function_results>
...
</function_results>
{antml:thinking}
...thinking about results
</antml:thinking> [6]
Interestingly, Claude is system-prompted to call at least 5 tools to answer a prompt when report or a deeper research is asked for
Complex queries using terms like "deep dive," "comprehensive," "analyze," "evaluate," "assess," "research," or "make a report" require AT LEAST 5 tool calls for thoroughness. [6]
circuit-tracer [3]
Anthropic released this week the circuit-tracer package. This tool is designed for researchers to look inside Large Language Models (LLMs) and better understand how they produce their outputs. It helps uncover the underlying computational steps that lead to specific behaviors. A key part of the method involves using something called cross-layer MLP transcoders. Think of transcoders as a special kind of tool that learns to represent the complex computations happening inside the model's MLP layers (which are usually hard to see clearly) in a simpler, more understandable way using "features". Because these transcoders are "cross-layer," features learned at one step can directly influence later steps across different parts of the model, making the path of information easier to trace. The package then uses these transcoder features to create "attribution graphs” which essentially map out the direct influence of input words and these interpretable features on other features and ultimately on the model's final output words. You can visualize these graphs on neuronpedia.org (even pruning them to manage complexity), and intervene by changing feature activations to see how the model's output change.
RAG is dead, long live agentic retrieval [9]
Agentic Retrieval represents a significant evolution from traditional, naive chunk retrieval in Retrieval Augmented Generation (RAG) systems. Moving beyond simply retrieving the top-k most similar document chunks, Agentic Retrieval employs an agent layer to intelligently determine the most appropriate method for accessing information based on the specific user query. This can involve routing the query to different retrieval modes within a single knowledge base, such as retrieving document chunks for specific information, retrieving entire files based on metadata (like filename), or retrieving files based on their general content. Furthermore, Agentic Retrieval systems can extend to multiple indices or knowledge bases, utilizing a composite retriever with an agent layer that selects the most relevant indices based on descriptive information. A fully agentic retrieval system integrates this intelligence at multiple layers, dynamically routing queries from selecting the right index down to choosing the optimal retrieval mode within that index. This approach is seen as the future of retrieval for sophisticated agent-based systems, as it provides the precise and relevant context needed for agents to operate effectively and autonomously. General AI agents are defined by their capabilities in reasoning, memory, and executing tasks, such as calling tools via protocols like MCP, which is leveraged in agentic retrieval for enhanced data access.
Paper Insights
Spurious Rewards: Rethinking Training Signals in RLVR [7]
The paper presents the surprising discovery that Reinforcement Learning with Verifiable Rewards (RLVR) can dramatically improve the mathematical reasoning of Qwen2.5-Math models even when the rewards are weak, flawed, or entirely "spurious", such as rewarding random outputs or incorrect answers. These performance gains are often comparable to those achieved with ground truth rewards. This counterintuitive finding leads to the hypothesis that RLVR, at least at the scale studied, isn't teaching new reasoning skills but is instead surfacing useful reasoning representations or abilities that the model learned during its pretraining phase. The authors identify "code reasoning" (generating Python code to aid thinking) as a key behavior in Qwen2.5-Math models that RLVR, even with spurious rewards, significantly amplifies, and this behavior is strongly predictive of performance. However, these same weak or spurious rewards often fail to yield similar gains or can degrade performance in other model families like Llama3 and OLMo2, which lack the same effective pre-existing priors like code reasoning. This aligns with the "Cake that is Intelligence" analogy from Yann LeCun, where pretraining forms the "bulk" or "base" of the model's capabilities, analogous to the baked sponge, while Reinforcement Learning is seen as the "cherry on top", primarily enabling superficial tuning rather than altering the core composition established during the initial "baking" (training) process. The "Spurious Rewards" paper's results suggest that RL (the "cherry") is most effective when the strong foundational capabilities ("base") are already present from pretraining, allowing RL to simply tune the model to better utilize or express those existing strengths, even with minimal or noisy feedback, rather than teaching the capabilities from scratch. The paper concludes with the critical suggestion that RLVR research should be validated on diverse models beyond just one family like Qwen, given how dependent the outcomes are on the model's pre-existing priors.
MCP Github Server vulnerability [2]
Invariantlabs found a security vulnerability when using the MCP server from Github. The vulnerability leads to the exposure of private information/code to a public repo. The prerequisites for this vulnerability are just having a public repo, a private repo, using the Github MCP server to manage these and a client with Auto-Allow (you don’t have to accept every action of the client). And this is how it works:
In the Issue, the user asks for the exposure of personal information of the User (possibly by querying the personal memory the client has of the User) in a README.md with all the names of the private repos. All these information were commited and pushed in the public repo for everybody to read (address, salary etc.).
Hype as Infrastructure [8]
I read the chapter “Hype as Infrastructure” from the book Reshuffle , which was preprinted in the Artificial Intelligence Made Simple Newsletter [8]. It redefines hype not merely as marketing or manipulation, but as a critical mechanism for coordination and infrastructure building in fragmented systems lacking centralized authority. The central argument is that in environments marked by high uncertainty and interdependence among actors—such as those surrounding emerging technologies like AI—hype can act as a form of "soft power" to solve coordination problems, align incentives, and trigger large-scale collective action towards ambitious goals, akin to the game theory concept of the Stag Hunt. The historical example of Renaissance Florence is used to illustrate this; despite lacking military "moats," Florence strategically invested in art, philosophy, and diplomacy to cultivate a narrative of civilizational value, making itself a focal point too culturally significant to attack, thus achieving generational defense through perception rather than physical barriers. In modern contexts, the success of Tesla's Supercharger network over a federally supported standard is presented as another example where narrative and infrastructure, fueled by a vision of energy revolution, succeeded in achieving market-wide coordination where traditional institutional methods failed. Hype, in this view, generates attention faster than fundamentals improve (temporal arbitrage), can redistribute risks and costs, and acts as "productive fiction" that helps solve coordination problems by painting a believable future. While acknowledging the potential for "bad" hype used for short-term gain, the source argues that "good" hype, focused on strategic coordination, is increasingly essential for progress where outcomes rely on early alignment across diverse actors.
Other
Japan just broke the world economy (worse than Greece) [12]
Yen Carry Trades are a major global financial strategy (that has been done for decades) where investors borrow money in Japanese yen at very low interest rates (or even zero interest, which were introduced in 1999 to stimulate economic growth), convert it to currencies with higher yields (dollar, euro, …), and invest that money overseas in various assets like stocks, bonds, and crypto. This difference in borrowing cost and investment return created significant profit opportunities. However, this trade depends on Japan's interest rates remaining low and the yen staying weak. Japan's interest rates have begun to rise (to fight inflation and because of demographic challenges) and the yen strengthened in 2024 and 2025, the Yen Carry Trade is starting to unwind, potentially putting trillions in global leverage at risk and creating ripple effects in markets worldwide.
Resources
[1] Vision as LoRA: https://arxiv.org/pdf/2503.20680
[2] MCP Github Server vulnerability: https://invariantlabs.ai/blog/mcp-github-vulnerability
[3] circuit-tracer package: https://github.com/safety-research/circuit-tracer
[4] Attribution-graphs: https://transformer-circuits.pub/2025/attribution-graphs/methods.html
[5] Anthropic System Prompts Release Notes: https://docs.anthropic.com/en/release-notes/system-prompts
[6] Github Repo of apparently the real full System Prompt: https://github.com/elder-plinius/CL4R1T4S/blob/main/ANTHROPIC/Claude_4.txt
[7] Spurious Rewards Paper: https://rethink-rlvr.notion.site/Spurious-Rewards-Rethinking-Training-Signals-in-RLVR-1f4df34dac1880948858f95aeb88872f
[8] Hype as Infrastructure: Artificial Intelligence Made Simple Newsletter May 28 2025
[9] RAG is dead, long live agentic retrieval: https://www.llamaindex.ai/blog/rag-is-dead-long-live-agentic-retrieval
[10] DeepSeek-R1-0528: https://huggingface.co/deepseek-ai/DeepSeek-R1-0528
[11] Reshuffle: https://www.amazon.com/Reshuffle-wins-restacks-knowledge-economy-ebook/dp/B0DTKW6NQV
[12] Japan just broke the world economy (worse than Greece) by Andrei Jikh (YT Video)