!pip install huggingface_hub
from huggingface_hub import notebook_login


notebook_login()

We install the huggingface_hub library and then import and invoke the notebook_login() function, which prompts you to authenticate your notebook session with your Hugging Face account, allowing you to seamlessly access and manage models, datasets, and other hub resources.

!pip uninstall -y torchvision


!pip install --upgrade torch torchvision --index-url https://download.pytorch.org/whl/cu118


!pip install --upgrade diffusers transformers accelerate safetensors gradio pillow

We first force-uninstalls any existing torchvision to clear potential conflicts, then reinstalls torch and torchvision from the CUDA 11.8–compatible PyTorch wheels, and finally upgrades key libraries, diffusers, transformers, accelerate, safetensors, gradio, and pillow, to ensure you have the latest versions for building and running GPU-accelerated generative pipelines and web demos.

import torch
from diffusers import StableDiffusionPipeline, StableDiffusion3Pipeline
import gradio as gr


device = "cuda" if torch.cuda.is_available() else "cpu"

We import PyTorch alongside both the Stable Diffusion v1 and v3 pipelines from the Diffusers library, as well as Gradio for building interactive demos. It then checks for CUDA availability and sets the device variable to “cuda” if a GPU is present; otherwise, it falls back to “cpu”, ensuring your models run on the optimal hardware.

pipe1 = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    safety_checker=None
).to(device)
pipe1.enable_attention_slicing()

We load the Stable Diffusion v1.5 model in half-precision (float16) without the built-in safety checker, transfers it to your selected device (GPU, if available), and then enables attention slicing to reduce peak VRAM usage during image generation.

pipe2 = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-base",
    torch_dtype=torch.float16,
    safety_checker=None
).to(device)
pipe2.enable_attention_slicing()

We load the Stable Diffusion v2 “base” model in 16-bit precision without the default safety filter, transfers it to your chosen device, and activates attention slicing to optimize memory usage during inference.

pipe3 = StableDiffusion3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-3-medium-diffusers",
    torch_dtype=torch.float16,
    safety_checker=None
).to(device)
pipe3.enable_attention_slicing()

We pull in Stability AI’s Stable Diffusion 3 “medium” checkpoint in 16-bit precision (skipping the built-in safety checker), transfers it to your selected device, and enables attention slicing to reduce GPU memory usage during generation.

def generate(prompt, steps, scale):
    img1 = pipe1(prompt, num_inference_steps=steps, guidance_scale=scale).images[0]
    img2 = pipe2(prompt, num_inference_steps=steps, guidance_scale=scale).images[0]
    img3 = pipe3(prompt, num_inference_steps=steps, guidance_scale=scale).images[0]
    return img1, img2, img3

Now, this function runs the same text prompt through all three loaded pipelines (pipe1, pipe2, pipe3) using the specified inference steps and guidance scale, then returns the first image from each, making it perfect for comparing outputs across Stable Diffusion v1.5, v2-base, and v3-medium.

def choose(selection):
    return f" You selected: **{selection}**"


with gr.Blocks() as demo:
    gr.Markdown("## AI Social-Post Generator with 3 Models")
    with gr.Row():
        prompt = gr.Textbox(label="Prompt", placeholder="A vibrant beach sunset…")
        steps  = gr.Slider( 1, 100, value=50, step=1,     label="Inference Steps")
        scale  = gr.Slider( 1.0, 20.0, value=7.5, step=0.1, label="Guidance Scale")
    btn = gr.Button("Generate Images")
    with gr.Row():
        out1 = gr.Image(label="Model 1: SD v1.5")
        out2 = gr.Image(label="Model 2: SD v2-base")
        out3 = gr.Image(label="Model 3: SD v3-medium")
    sel = gr.Radio(
        ["Model 1: SD v1.5","Model 2: SD v2-base","Model 3: SD v3-medium"],
        label="Select your favorite"
    )
    txt = gr.Markdown()


    btn.click(fn=generate, inputs=[prompt, steps, scale], outputs=[out1, out2, out3])
    sel.change(fn=choose, inputs=sel, outputs=txt)


demo.launch(share=True)

Finally, this Gradio app builds a three-column UI where you can enter a text prompt, adjust inference steps and guidance scale, then generate and display images from SD v1.5, v2-base, and v3-medium side by side. It also features a radio selector, allowing you to select your preferred model output, and displays a simple confirmation message when a choice is made.

In conclusion, by integrating Stability AI’s state-of-the-art diffusion architectures into an easy-to-use Gradio app, you’ve seen how effortlessly you can prototype, compare, and deploy stunning visuals that resonate on today’s platforms. From A/B-testing creative directions to automating campaign assets at scale, Stability AI provides the performance, flexibility, and vibrant community support to transform your content pipeline.

Check out the Colab Notebook. Don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit. For Promotion and Partnerships, please talk us.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post A Coding Guide to Compare Three Stability AI Diffusion Models (v1.5, v2-Base & SD3-Medium) Diffusion Capabilities Side-by-Side in Google Colab Using Gradio appeared first on MarkTechPost.

Addressing Disconnection and Autonomy Challenges

Unlike terrestrial applications, AI systems deployed in orbit face strict constraints—limited compute resources, constrained bandwidth, and high-latency communication links with ground stations. Space Llama has been designed to function entirely offline, allowing astronauts to access technical assistance, documentation, and maintenance protocols without requiring live support from mission control.

To address these constraints, the AI model had to be optimized for onboard deployment, incorporating the ability to reason over mission-specific queries, retrieve context from local data stores, and interact with astronauts in natural language—all without internet connectivity.

Technical Framework and Integration Stack

The deployment leverages a combination of commercially available and mission-adapted technologies:

• Llama 3.2: Meta’s latest open-source LLM serves as the foundation, fine-tuned for contextual understanding and general reasoning tasks in edge environments. Its open architecture enables modular adaptation for aerospace-grade applications.
• A2E2 (AI for Edge Environments): Booz Allen’s AI framework provides containerized deployment and modular orchestration tailored to constrained environments like the ISS. It abstracts complexity in model serving and resource allocation across diverse compute layers.
• HPE Spaceborne Computer-2: This edge computing platform, developed by Hewlett Packard Enterprise, provides reliable high-performance processing hardware for space. It supports real-time inference workloads and model updates when necessary.
• NVIDIA CUDA-capable GPUs: These enable the accelerated execution of transformer-based inference tasks while staying within the ISS’s strict power and thermal budgets.

This integrated stack ensures that the model operates within the limits of orbital infrastructure, delivering utility without compromising reliability.

Open-Source Strategy for Aerospace AI

The selection of an open-source model like Llama 3.2 aligns with growing momentum around transparency and adaptability in mission-critical AI. The benefits include:

• Modifiability: Engineers can tailor the model to meet specific operational requirements, such as natural language understanding in mission terminology or handling multi-modal astronaut inputs.
• Data Sovereignty: With all inference running locally, sensitive data never needs to leave the ISS, ensuring compliance with NASA and partner agency privacy standards.
• Resource Optimization: Open access to the model’s architecture allows for fine-grained control over memory and compute use—critical for environments where system uptime and resilience are prioritized.
• Community-Based Validation: Using a widely studied open-source model promotes reproducibility, transparency in behavior, and better testing under mission simulation conditions.

Toward Long-Duration and Autonomous Missions

Space Llama is not just a research demonstration—it lays the groundwork for embedding AI systems into longer-term missions. In future scenarios like lunar outposts or deep-space habitats, where round-trip communication latency with Earth spans minutes or hours, onboard intelligent systems must assist with diagnostics, operations planning, and real-time problem-solving.

Furthermore, the modular nature of Booz Allen’s A2E2 platform opens up the potential for expanding the use of LLMs to non-space environments with similar constraints—such as polar research stations, underwater facilities, or forward operating bases in military applications.

Conclusion

The Space Llama initiative represents a methodical advancement in deploying AI systems to operational environments beyond Earth. By combining Meta’s open-source LLMs with Booz Allen’s edge deployment expertise and proven space computing hardware, the collaboration demonstrates a viable approach to AI autonomy in space.

Rather than aiming for generalized intelligence, the model is engineered for bounded, reliable utility in mission-relevant contexts—an important distinction in environments where robustness and interpretability take precedence over novelty.

As space systems become more software-defined and AI-assisted, efforts like Space Llama will serve as reference points for future AI deployments in autonomous exploration and off-Earth habitation.

Check out the Details here. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Meta and Booz Allen Deploy Space Llama: Open-Source AI Heads to the ISS for Onboard Decision-Making appeared first on MarkTechPost.

One challenge in building reasoning-focused models is achieving strong, simultaneous performance in mathematics and programming while maintaining a relatively small model size. Most competitive results in these domains are achieved by models with approximately 32 billion parameters or more. These large models are often used because smaller ones struggle with generalization and reward optimization in reinforcement learning tasks, particularly when it comes to code-based problem-solving. Sparse reward feedback, limited high-quality data, and weak base model architecture make it difficult to develop compact yet powerful models. Additionally, the data used to train these models is not always curated with reasoning in mind, often resulting in training inefficiencies and limited gains in problem-solving abilities.

To address reasoning challenges, several models, including OpenAI’s o-series, DeepSeek R1, and Claude 3.7, have been introduced, leveraging massive parameter counts and complex reinforcement learning strategies. These models employ techniques such as step-by-step planning and backtracking to enhance reasoning, particularly in algorithmic thinking and math-related tasks. However, they heavily depend on post-training stages and underplay the importance of high-quality pre-training data. Many also rely on fixed template-based reward systems that are prone to reward hacking. Code generation benchmarks often reveal that these models perform inconsistently in challenging tasks due to shallow pretraining foundations and ineffective reward signal modeling during fine-tuning.

A research team from Xiaomi introduced the MiMo-7B family of language models with a focused approach to overcoming these barriers. The innovation lies in treating both pre-training and post-training as equally critical phases for developing reasoning capabilities. The base model, MiMo-7B-Base, was trained from scratch using a dataset comprising 25 trillion tokens. This dataset was constructed with a three-stage mixture strategy that progressively increased the share of mathematical and programming content. An additional multiple-token prediction (MTP) objective was introduced during pre-training to improve both performance and inference speed. For post-training, the team developed a curated dataset of 130,000 verifiable math and programming problems, each tagged with difficulty scores. Reinforcement learning was then applied using a difficulty-driven reward framework, allowing more nuanced and effective feedback during training. This resulted in two major variants: MiMo-7B-RL and MiMo-7B-RL-Zero.

The pre-training methodology started by extracting reasoning-heavy content from web pages, academic papers, and books using a custom HTML extraction tool designed to preserve math equations and code snippets. Unlike generic pipelines, this extractor retained structural elements critical to problem-solving domains. The team then enhanced the PDF parsing tools to interpret scientific and programming content accurately. To prevent data duplication, global deduplication was applied using URL-based and MinHash techniques. The training corpus was filtered using small language models fine-tuned to tag content quality, replacing outdated heuristic-based filters that often removed valuable reasoning examples. High-quality synthetic reasoning data was also generated from advanced models and added in the final stage of training. This three-stage approach resulted in a final training mix comprising 70% math and code data in stage two and an additional 10% of synthetic content in stage three. The maximum context length was extended from 8,192 to 32,768 tokens, ensuring the model could handle long-form reasoning problems.

In the reinforcement learning stage, the research team engineered a seamless rollout engine to accelerate training and validation. This infrastructure incorporated asynchronous reward computation and early termination mechanisms to reduce GPU idle time, resulting in 2.29 times faster training and 1.96 times faster validation. The model’s policy was optimized using fine-grained rewards derived from the difficulty of test cases, addressing the sparse reward issue in programming benchmarks. Data re-sampling techniques were introduced to maintain training stability and increase rollout sampling efficiency. These strategies collectively enabled the MiMo-7B variants to learn effectively, even from cold-start states where no pre-fine-tuned initialization is available.

Performance evaluation revealed that MiMo-7B-Base achieved a score of 75.2 on the Big-Bench Hard (BBH) task, surpassing other open-source 7B models. It also performed well on SuperGPQA, which includes graduate-level reasoning questions. The post-trained MiMo-7B-RL scored 55.4 on the AIME 2025 benchmark, surpassing OpenAI’s o1-mini by 4.7 points. On code generation tasks, it outperformed much larger models like DeepSeek-R1-Zero-32B and Qwen2.5-32B-RL-Zero on both LiveCodeBench v5 and v6. These results demonstrate that a properly optimized 7B model can rival or even outperform models with more than four times the number of parameters.

The MiMo-7B project serves as a concrete demonstration of how pre-training, data quality, and reinforcement learning infrastructure contribute to the final reasoning capability of a language model. By rethinking the pipeline from data extraction to reward computation, the Xiaomi research team achieved compact yet powerful models suitable for real-world applications in mathematics, coding, and logic. Their approach highlights the untapped potential of small models and challenges the assumption that size alone determines intelligence or versatility.

Key Takeaways from the Research on MiMo-7B:  

1. MiMo-7B was trained on a massive dataset of 25 trillion tokens, targeting reasoning tasks through the use of structured data mixtures.  
2. 130,000 math and code problems were used in RL training, each annotated with difficulty scores to enable effective reward shaping.  
3. Three-stage pre-training raised math and coding content to 70%, followed by 10% synthetic problem-solving data.  
4. A seamless rollout engine increased RL training speed by 2.29 times and validation by 1.96 times.  
5. MiMo-7B-RL achieved 55.4 on AIME 2025, outperforming OpenAI o1-mini by 4.7 points.  
6. MiMo-7B models are publicly available and include all checkpoints: base, SFT, and RL variants.  
7. The model’s success shows that small, well-designed models can rival or exceed the performance of 32B models in reasoning tasks.  

Check out the Paper and GitHub Page. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Xiaomi introduced MiMo-7B: A Compact Language Model that Outperforms Larger Models in Mathematical and Code Reasoning through Rigorous Pre-Training and Reinforcement Learning appeared first on MarkTechPost.

!pip install -qU langchain-dappier langchain langchain-openai langchain-community langchain-core openai

We bootstrap our Colab environment by installing the core LangChain libraries, both the Dappier extensions and the community integrations, alongside the official OpenAI client. With these packages in place, we will have seamless access to Dappier’s real-time search and recommendation tools, the latest LangChain runtimes, and the OpenAI API, all in one environment.

import os
from getpass import getpass


os.environ["DAPPIER_API_KEY"] = getpass("Enter our Dappier API key: ")


os.environ["OPENAI_API_KEY"] = getpass("Enter our OpenAI API key: ")

We securely capture our Dappier and OpenAI API credentials at runtime, thereby avoiding the hard-coding of sensitive keys in our notebook. By using getpass, the prompts ensure our inputs remain hidden, and setting them as environment variables makes them available to all subsequent cells without exposing them in logs.

from langchain_dappier import DappierRealTimeSearchTool


search_tool = DappierRealTimeSearchTool()
print("Real-time search tool ready:", search_tool)

We import Dappier’s real‐time search module and create an instance of the DappierRealTimeSearchTool, enabling our notebook to execute live web queries. The print statement confirms that the tool has been initialized successfully and is ready to handle search requests.

from langchain_dappier import DappierAIRecommendationTool


recommendation_tool = DappierAIRecommendationTool(
    data_model_id="dm_01j0pb465keqmatq9k83dthx34",
    similarity_top_k=3,
    ref="sportsnaut.com",
    num_articles_ref=2,
    search_algorithm="most_recent",
)
print("Recommendation tool ready:", recommendation_tool)

We set up Dappier’s AI-powered recommendation engine by specifying our custom data model, the number of similar articles to retrieve, and the source domain for context. The DappierAIRecommendationTool instance will now use the “most_recent” algorithm to pull in the top-k relevant articles (here, two) from our specified reference, ready for query-driven content suggestions.

from langchain.chat_models import init_chat_model


llm = init_chat_model(
    model="gpt-3.5-turbo",
    model_provider="openai",
    temperature=0,
)
llm_with_tools = llm.bind_tools([search_tool])
print(" llm_with_tools ready")

We create an OpenAI chat model instance using gpt-3.5-turbo with a temperature of 0 to ensure consistent responses, and then bind the previously initialized search tool so that the LLM can invoke real-time searches. The final print statement confirms that our LLM is ready to call Dappier’s tools within our conversational flows.

import datetime
from langchain_core.prompts import ChatPromptTemplate


today = datetime.datetime.today().strftime("%Y-%m-%d")
prompt = ChatPromptTemplate([
    ("system", f"we are a helpful assistant. Today is {today}."),
    ("human", "{user_input}"),
    ("placeholder", "{messages}"),
])


llm_chain = prompt | llm_with_tools
print(" llm_chain built")

We construct the conversational “chain” by first building a ChatPromptTemplate that injects the current date into a system prompt and defines slots for user input and prior messages. By piping the template (|) into our llm_with_tools, we create an llm_chain that automatically formats prompts, invokes the LLM (with real-time search capability), and handles responses in a seamless workflow. The final print confirms the chain is ready to drive end-to-end interactions.

from langchain_core.runnables import RunnableConfig, chain


@chain
def tool_chain(user_input: str, config: RunnableConfig):
    ai_msg = llm_chain.invoke({"user_input": user_input}, config=config)
    tool_msgs = search_tool.batch(ai_msg.tool_calls, config=config)
    return llm_chain.invoke(
        {"user_input": user_input, "messages": [ai_msg, *tool_msgs]},
        config=config
    )


print(" tool_chain defined")

We define an end-to-end tool_chain that first sends our prompt to the LLM (capturing any requested tool calls), then executes those calls via search_tool.batch, and finally feeds both the AI’s initial message and the tool outputs back into the LLM for a cohesive response. The @chain decorator transforms this into a single, runnable pipeline, allowing us to simply call tool_chain.invoke(…) to handle both thinking and searching in a single step.

res = search_tool.invoke({"query": "What happened at the last Wrestlemania"})
print(" Search:", res)

We demonstrate a direct query to Dappier’s real-time search engine, asking “What happened at the last WrestleMania,” and immediately print the structured result. It shows how easily we can leverage search_tool.invoke to fetch up-to-the-moment information and inspect the raw response in our notebook.

rec = recommendation_tool.invoke({"query": "latest sports news"})
print(" Recommendation:", rec)


out = tool_chain.invoke("Who won the last Nobel Prize?")
print(" Chain output:", out)

Finally, we showcase both our recommendation and full-chain workflows in action. First, it calls recommendation_tool.invoke with “latest sports news” to fetch relevant articles from our custom data model, then prints those suggestions. Next, it runs the tool_chain.invoke(“Who won the last Nobel Prize?”) to perform an end-to-end LLM query combined with real-time search, printing the AI’s synthesized answer, and integrating live data.

In conclusion, we now have a robust baseline for embedding Dappier AI capabilities into any conversational workflow. We’ve seen how effortlessly Dappier’s real-time search empowers our LLM to access fresh facts, while the recommendation tool enables us to deliver contextually relevant insights from proprietary data sources. From here, we can customize search parameters (e.g., refining query filters) or fine-tune recommendation settings (e.g., adjusting similarity thresholds and reference domains) to suit our domain.

Check out the Dappier Platform and Notebook here. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post A Step-by-Step Coding Guide to Integrate Dappier AI’s Real-Time Search and Recommendation Tools with OpenAI’s Chat API appeared first on MarkTechPost.

Solving the Comprehension Bottleneck

In research, business, and education, one of the consistent challenges is information overload. While large language models (LLMs) like Gemini can generate fluent summaries, accessibility and modality gaps still limit their practical utility—especially for non-native English speakers, visually impaired users, or individuals who prefer auditory content over text. Google addresses this with Audio Overviews: human-like spoken summaries automatically generated from user-supplied source materials.

This expansion aims to solve both linguistic and modal bottlenecks simultaneously, helping users engage with dense material more flexibly. Whether it’s an academic journal, business strategy deck, or a long PDF manual, users can now consume synthesized summaries in their preferred language and format.

A Multilingual, Multi-Modal Summarization Framework

Audio Overviews are not mere text-to-speech (TTS) features. They represent an integrated summarization pipeline:

1. Grounded Content Understanding: NotebookLM uses Google’s Gemini language model to analyze and extract relevant information from uploaded documents.
2. Topic Modeling: The system segments information into digestible chunks, choosing what’s most important based on user queries or default salience heuristics.
3. Natural Speech Generation: Using Google’s WaveNet and multilingual speech synthesis models, it generates lifelike audio in 50+ languages including French, Hindi, Japanese, German, Portuguese, Arabic, Swahili, and more.
4. Contextual Learning: Audio Overviews are not static; they evolve based on user interactions. Follow-up questions can be asked in any supported language, allowing continuous learning across text and voice modalities.

What differentiates Audio Overviews from simple TTS pipelines is the blend of summarization, topic selection, and fluent narrative construction—especially across diverse languages with varying grammatical and phonetic rules.

Technical Enhancements and Accessibility Focus

NotebookLM’s multilingual support is built upon Google’s foundational language and speech platforms, including Gemini 1.5, TTS Research (Tacotron, WaveNet), and Translate models. The system dynamically adjusts the speech output based on regional pronunciation norms and cultural context.

To ensure equitable access, Google also made the audio outputs downloadable and compatible with screen readers, mobile devices, and offline playback apps. This makes the tool especially valuable for students and researchers in lower-bandwidth regions.

Early user feedback has indicated notable satisfaction with the clarity and fidelity of summaries. For example, in pilot deployments across educational institutions in India and Germany, students reported a 40% faster comprehension rate when consuming audio summaries compared to reading full documents.

Implications for Global Learning and Enterprise Use

The launch positions NotebookLM as more than a note-taking or summarization tool—it is evolving into an AI-powered research assistant that adapts to global, multimodal workflows. From corporate teams collaborating across continents to academic researchers conducting multilingual literature reviews, the new capabilities significantly lower the barrier to deep content engagement.

For businesses, this opens up new possibilities in training, onboarding, compliance, and multilingual support content. For education, it enables inclusive learning environments that support auditory learners and underserved language communities.

What’s Next?

Google confirms that additional language support is already in development. Furthermore, future updates may include speaker customization, tonal adjustments (e.g., formal vs. casual), and integration with platforms like Google Docs, YouTube transcripts, and Chrome extensions.

Check out the Official Blog. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Google NotebookLM Launches Audio Overviews in 50+ Languages, Expanding Global Accessibility for AI Summarization appeared first on MarkTechPost.

A key challenge in sequential modeling is dynamically controlling memory. Standard attention-based models, such as the Transformer, process and store all input information uniformly, regardless of its relevance over time. This approach can be suboptimal when recent inputs carry more significance for a task, or when older inputs introduce noise. Traditional recurrent models address this with mechanisms such as forget gates, which allow them to modulate memory retention. However, these models struggle to maintain performance over extended sequences because of their fixed-size hidden states. The Transformer, while powerful, lacks a native method for discarding less useful past information in a context-sensitive manner. As a result, tasks that demand selective memory can suffer, especially when input lengths grow substantially and noise accumulates.

To address memory challenges, some strategies have introduced static positional biases into attention mechanisms. For instance, ALiBi adds predefined slopes to attention logits to simulate a form of recency weighting. However, such methods lack adaptability, as they do not consider the content of the input when deciding what to retain. Other efforts, such as Mamba-2 and GLA, implement gating within linear attention frameworks but often sacrifice normalization, a key aspect of Transformer accuracy. Also, these models tend to deviate significantly from the Transformer structure, making them less compatible with Transformer-based optimizations and pretraining paradigms. Thus, a gap remains for an approach that can dynamically forget in a learnable and efficient manner while preserving the Transformer’s computational strengths.

Researchers from Mila & Universite de Montreal and MakerMaker AI proposed a novel architecture called the Forgetting Transformer (FoX). This model introduces a mechanism known as Forgetting Attention, which inserts a scalar forget gate into the softmax attention process. Unlike existing recurrent models, this modification is fully compatible with parallel computation and avoids the need for positional embeddings. The forget gate adjusts the raw attention scores based on the data itself, allowing FoX to effectively down-weight less relevant past inputs. Importantly, the model retains full compatibility with the efficient FlashAttention algorithm, ensuring minimal deployment overhead. Two architectural variants were tested: FoX, based on LLaMA, and FoX (Pro), which incorporates normalization techniques and token-shifting mechanisms derived from recent recurrent models.

Technically, the model computes forget gate values for each timestep using a sigmoid activation on a learned linear transformation of the input. These scalar gate values are then used to bias attention logits through a log-sum formulation, modifying the softmax operation in a hardware-efficient manner. The modification is implemented by computing the cumulative sum of log forget values and adjusting attention weights without requiring the instantiation of large matrices. Multi-head attention support is retained, with each head maintaining independent forget gate parameters. The Pro variant introduces output normalization and output gates, along with a key-value shift mechanism that mixes current and previous tokens in a learnable manner. These adjustments further refine context sensitivity and model flexibility without significantly increasing the number of parameters.

In a long-context language modeling task using the LongCrawl64 dataset (a 48-billion-token subset of RedPajama-v2), FoX consistently surpassed both standard Transformer baselines and leading recurrent models. Per-token loss metrics showed a sharper decline for FoX across token positions, indicating better context utilization. At position 64,000, FoX (Pro) achieved significantly lower loss values than Transformer (Pro) and LLaMA variants. Also, perplexity evaluations demonstrated that FoX maintains robust accuracy across increasing validation context lengths, with performance degrading less sharply beyond the training limit of 16,384 tokens. Competing models, such as Mamba-2 and DeltaNet, showed earlier plateaus, highlighting FoX’s superior extrapolation capabilities. Training was performed with 760 million parameters using the TikToken tokenizer for GPT-2, with extensive tuning for learning rates and head dimensions. Fox preferred higher learning rates and smaller head dimensions, indicating architectural resilience and adaptability.

The researchers emphasized that Forgetting Attention retains the core benefits of the Transformer while overcoming its limitations regarding selective memory. They demonstrated that the forget gate introduces a data-driven recency bias that strengthens performance in both short and long sequences. Additionally, the implementation incurs minimal computational cost and requires no additional memory overhead, thanks to its compatibility with FlashAttention. Notably, Forgetting Attention also generalizes static biases, such as ALiBi, by introducing learnable gates, providing evidence that dynamic biasing is significantly more effective. FoX models also matched or exceeded standard Transformer performance on downstream tasks, with the Pro variant showing consistent superiority, especially in functions that reward adaptability across contexts.

This work demonstrates that the effective integration of dynamic memory mechanisms into Transformer architectures is not only feasible but also beneficial across a wide range of benchmarks. The introduction of a forget gate within the attention computation allows models to discard irrelevant information in a learned manner, substantially improving focus and generalization. The compatibility with high-performance implementations, such as FlashAttention, ensures that such improvements come without trade-offs in efficiency.

Several Key takeaways from the research on FoX include:

• FoX introduces Forgetting Attention, enhancing standard softmax attention with learnable forget gates.
• Two architectural variants were tested: FoX (LLaMA) and FoX (Pro), with the latter incorporating additional normalization and gating layers.
• FoX models trained on 48B tokens with 760M parameters significantly outperformed Transformers in long-context modeling.
• Per-token loss L(i) and perplexity P(l) confirmed that FoX maintained low error rates even beyond 64k-token sequences.
• Forgetting Attention is a generalization of ALiBi, offering dynamic, data-dependent gating over fixed biases.
• The Pro architecture further improved results with minimal overhead by using output normalization and token shift mechanisms.
• Hardware compatibility was preserved through modifications to FlashAttention, enabling practical deployment at scale.

Check out the Paper and Code. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Mila & Universite de Montreal Researchers Introduce the Forgetting Transformer (FoX) to Boost Long-Context Language Modeling without Sacrificing Efficiency appeared first on MarkTechPost.

What sets Crawl4AI apart is its unified API, which seamlessly switches between browser-based (Playwright) and HTTP-only strategies, its robust error-handling hooks, and its declarative extraction schemas. Unlike traditional headless-browser workflows, Crawl4AI allows you to choose the most lightweight and performant backend, making it ideal for scalable data pipelines, on-the-fly ETL in notebooks, or feeding LLMs and analytics tools with clean JSON/CSV outputs.

!pip install -U crawl4ai httpx

First, we install (or upgrade) Crawl4AI, the core asynchronous crawling framework, alongside HTTPX. This high-performance HTTP client provides all the building blocks we need for lightweight, asynchronous web scraping directly in Colab.

import asyncio, json, pandas as pd
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, HTTPCrawlerConfig
from crawl4ai.async_crawler_strategy import AsyncHTTPCrawlerStrategy
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy

We bring in Python’s core async and data‑handling modules, asyncio for concurrency, json for parsing, and pandas for tabular storage, alongside Crawl4AI’s essentials: AsyncWebCrawler to drive the crawl, CrawlerRunConfig and HTTPCrawlerConfig to configure extraction and HTTP settings, AsyncHTTPCrawlerStrategy for a browser‑free HTTP backend, and JsonCssExtractionStrategy to map CSS selectors into structured JSON.

http_cfg = HTTPCrawlerConfig(
    method="GET",
    headers={
        "User-Agent":      "crawl4ai-bot/1.0",
        "Accept-Encoding": "gzip, deflate"
    },
    follow_redirects=True,
    verify_ssl=True
)
crawler_strategy = AsyncHTTPCrawlerStrategy(browser_config=http_cfg)

Here, we instantiate an HTTPCrawlerConfig to define our HTTP crawler’s behavior, using a GET request with a custom User-Agent, gzip/deflate encoding only, automatic redirects, and SSL verification. We then plug that into AsyncHTTPCrawlerStrategy, allowing Crawl4AI to drive the crawl via pure HTTP calls rather than a full browser.

schema = {
    "name": "Quotes",
    "baseSelector": "div.quote",
    "fields": [
        {"name": "quote",  "selector": "span.text",      "type": "text"},
        {"name": "author", "selector": "small.author",   "type": "text"},
        {"name": "tags",   "selector": "div.tags a.tag", "type": "text"}
    ]
}
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=False)
run_cfg = CrawlerRunConfig(extraction_strategy=extraction_strategy)

We define a JSON‑CSS extraction schema targeting each quote block (div.quote) and its child elements (span.text, small.author, div.tags a.tag), then initializes a JsonCssExtractionStrategy with that schema, and wraps it in a CrawlerRunConfig so Crawl4AI knows exactly what structured data to pull on each request.

async def crawl_quotes_http(max_pages=5):
    all_items = []
    async with AsyncWebCrawler(crawler_strategy=crawler_strategy) as crawler:
        for p in range(1, max_pages+1):
            url = f"https://quotes.toscrape.com/page/{p}/"
            try:
                res = await crawler.arun(url=url, config=run_cfg)
            except Exception as e:
                print(f" Page {p} failed outright: {e}")
                continue


            if not res.extracted_content:
                print(f" Page {p} returned no content, skipping")
                continue


            try:
                items = json.loads(res.extracted_content)
            except Exception as e:
                print(f" Page {p} JSON‑parse error: {e}")
                continue


            print(f" Page {p}: {len(items)} quotes")
            all_items.extend(items)


    return pd.DataFrame(all_items)

Now, this asynchronous function orchestrates the HTTP‑only crawl: it spins up an AsyncWebCrawler with our AsyncHTTPCrawlerStrategy, iterates through each page URL, and safely awaits crawler.arun(), handles any request or JSON parsing errors and collects the extracted quote records into a single pandas DataFrame for downstream analysis.

df = asyncio.get_event_loop().run_until_complete(crawl_quotes_http(max_pages=3))
df.head()

Finally, we kick off the crawl_quotes_http coroutine on Colab’s existing asyncio loop, fetching three pages of quotes, and then display the first few rows of the resulting pandas DataFrame to verify that our crawler returned structured data as expected.

In conclusion, by combining Google Colab’s zero-config environment with Python’s asynchronous ecosystem and Crawl4AI’s flexible crawling strategies, we have now developed a fully automated pipeline for scraping and structuring web data in minutes. Whether you need to spin up a quick dataset of quotes, build a refreshable news‑article archive, or power a RAG workflow, Crawl4AI’s blend of httpx, asyncio, JsonCssExtractionStrategy, and AsyncHTTPCrawlerStrategy delivers both simplicity and scalability. Beyond pure HTTP crawls, you can instantly pivot to Playwright‑driven browser automation without rewriting your extraction logic, underscoring why Crawl4AI stands out as the go‑to framework for modern, production‑ready web data extraction.

Here is the Colab Notebook. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post A Coding Guide to Asynchronous Web Data Extraction Using Crawl4AI: An Open-Source Web Crawling and Scraping Toolkit Designed for LLM Workflows appeared first on MarkTechPost.

In recent years, the phenomenon of grokking—where deep learning models exhibit a delayed yet sudden transition from memorization to generalization—has prompted renewed investigation into training dynamics. Initially observed in small algorithmic tasks like modular arithmetic, grokking reveals that models can reach near-perfect training accuracy while validation performance remains poor for a prolonged period. Eventually, and often abruptly, the model begins to generalize. Understanding what governs this transition is important not just for interpretability, but also for optimizing training efficiency in deep networks. Prior studies have highlighted the role of weight decay and regularization. However, the specific influence of optimizers on this process has been underexplored.

Investigating Optimizer Effects on Grokking

This AI paper from Microsoft examines the impact of optimizer choice on grokking behavior. Specifically, it contrasts the performance of the widely adopted AdamW optimizer with Muon, a newer optimization algorithm that incorporates spectral norm constraints and second-order information. The study investigates whether these features enable Muon to expedite the generalization phase.

The experiments span seven algorithmic tasks—primarily modular arithmetic operations and parity classification—using a modern Transformer architecture. Each task is designed to reliably exhibit grokking under appropriate training conditions. The research also includes a comparative analysis of softmax variants (standard softmax, stablemax, and sparsemax) to evaluate whether output normalization plays a secondary role in modulating training dynamics. However, the core investigation centers on the optimizer.

Architectural and Optimization Design

The underlying model architecture adopts standard Transformer components, implemented in PyTorch. It includes multi-head self-attention, rotary positional embeddings (RoPE), RMS normalization, SiLU activations, and dropout-based regularization. Input tokens—numerical values or operators—are encoded through simple identity embeddings.

The key distinction lies in the optimizer behavior:

• AdamW, a baseline in contemporary deep learning workflows, uses adaptive learning rates with decoupled weight decay.
• Muon, in contrast, applies orthogonalized gradients, enforces spectral norm constraints to stabilize training, and approximates second-order curvature for more informative updates.

These mechanisms are intended to promote broader exploration during optimization, mitigate instability (e.g., “softmax collapse”), and synchronize learning progress across layers. Muon’s ability to regulate update magnitude in accordance with layer dimensions is particularly relevant in avoiding inefficient memorization pathways.

Three softmax configurations—Softmax, Stablemax, and Sparsemax—are included to assess whether numerical stability or sparsity of the output distribution influences grokking. This helps ensure that the observed effects stem primarily from optimizer dynamics rather than output activation nuances.

Empirical Evaluation and Results

The study’s empirical protocol is methodically designed. Each optimizer-softmax-task combination is evaluated across multiple seeds to ensure statistical robustness. Grokking is operationally defined as the first epoch where validation accuracy surpasses 95% following training accuracy stabilization.

The results indicate a consistent and statistically significant advantage for Muon. On average, Muon reaches the grokking threshold in 102.89 epochs, compared to 153.09 epochs for AdamW. This difference is not only numerically large but also statistically rigorous (t = 5.0175, p ≈ 6.33e−8). Additionally, Muon demonstrates a tighter distribution of grokking epochs across all conditions, suggesting more predictable training trajectories.

All tasks were conducted on NVIDIA H100 GPUs using a unified codebase and standardized configurations. Tasks include modular addition, multiplication, division, exponentiation, GCD, and a 10-bit parity task. Dataset sizes ranged from 1,024 to 9,409 examples, with training-validation splits adjusted per task to maintain consistency.

Conclusion

The findings provide strong evidence that optimizer geometry significantly influences the emergence of generalization in overparameterized models. By steering the optimization path through second-order-aware updates and spectral norm constraints, Muon appears to facilitate a more direct route toward discovering the underlying data structure, bypassing prolonged overfitting phases.

This study underscores the broader need to consider optimization strategy as a first-class factor in neural training design. While prior work emphasized data and regularization, these results suggest that optimizer architecture itself can play a pivotal role in shaping training dynamics.

Check out the Paper. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Muon Optimizer Significantly Accelerates Grokking in Transformers: Microsoft Researchers Explore Optimizer Influence on Delayed Generalization appeared first on MarkTechPost.

Technical Overview and Model Capabilities

Dia is designed for high-fidelity speech synthesis, incorporating a transformer-based architecture that balances expressive prosody modeling with computational efficiency. The model supports zero-shot voice cloning, enabling it to replicate a speaker’s voice from a short reference audio clip. Unlike traditional systems that require fine-tuning for each new speaker, Dia generalizes effectively across voices without retraining.

A notable technical feature of Dia is its ability to synthesize non-verbal vocalizations, such as coughing and laughter. These components are typically excluded from many standard TTS systems, yet they are critical for generating naturalistic and contextually rich audio. Dia models these sounds natively, contributing to more human-like speech output.

The model also supports real-time synthesis, with optimized inference pipelines allowing it to operate on consumer-grade devices, including MacBooks. This performance characteristic is particularly valuable for developers seeking low-latency deployment without relying on cloud-based GPU servers.

Deployment and Licensing

Dia’s release under the Apache 2.0 license offers broad flexibility for both commercial and academic use. Developers can fine-tune the model, adapt its outputs, or integrate it into larger voice-based systems without licensing constraints. The training and inference pipeline is written in Python and integrates with standard audio processing libraries, lowering the barrier to adoption.

The model weights are available directly via Hugging Face, and the repository provides a clear setup process for inference, including examples of input text-to-audio generation and voice cloning. The design favors modularity, making it easy to extend or customize components such as vocoders, acoustic models, or input preprocessing.

Comparisons and Initial Reception

While formal benchmarks have not been extensively published, preliminary evaluations and community tests suggest that Dia performs comparably—if not favorably—to existing commercial systems in areas such as speaker fidelity, audio clarity, and expressive variation. The inclusion of non-verbal sound support and open-source availability further distinguishes it from its proprietary counterparts.

Since its release, Dia has gained significant attention within the open-source AI community, quickly reaching the top ranks on Hugging Face’s trending models. The community response highlights the growing demand for accessible, high-performance speech models that can be audited, modified, and deployed without platform dependencies.

Broader Implications

The release of Dia fits within a broader movement toward democratizing advanced speech technologies. As TTS applications expand—from accessibility tools and audiobooks to interactive agents and game development—the availability of open, high-quality voice models becomes increasingly important.

By releasing Dia with an emphasis on usability, performance, and transparency, Nari Labs contributes meaningfully to the TTS research and development ecosystem. The model provides a strong baseline for future work in zero-shot voice modeling, multi-speaker synthesis, and real-time audio generation.

Conclusion

Dia represents a mature and technically sound contribution to the open-source TTS space. Its ability to synthesize expressive, high-quality speech—including non-verbal audio—combined with zero-shot cloning and local deployment capabilities, makes it a practical and adaptable tool for developers and researchers alike. As the field continues to evolve, models like Dia will play a central role in shaping more open, flexible, and efficient speech systems.

Check out the Model on Hugging Face, GitHub Page and Demo. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Open-Source TTS Reaches New Heights: Nari Labs Releases Dia, a 1.6B Parameter Model for Real-Time Voice Cloning and Expressive Speech Synthesis on Consumer Device appeared first on MarkTechPost.

Physical Intelligence Introduces π-0.5: A Framework for Embodied Adaptation

To address these constraints, Physical Intelligence has introduced π-0.5—a lightweight and modular framework designed to integrate perception, control, and learning directly within physical systems. As described in their recent blog post, π-0.5 serves as a foundational building block for what the team terms “physical intelligence”: systems that learn from and adapt to the physical world through constant interaction, not abstraction alone.

Rather than isolating intelligence in a centralized digital core, π-0.5 distributes processing and control throughout the system in compact modules. Each module, termed a “π-node,” encapsulates sensor inputs, local actuation logic, and a small, trainable neural component. These nodes can be chained or scaled across various embodiments, from wearables to autonomous agents, and are designed to react locally before resorting to higher-level computation. This architecture reflects a core assumption of the Physical Intelligence team: cognition emerges from action—not apart from it.

Technical Composition and Functional Characteristics

π-0.5 combines three core elements: (1) low-latency signal processing, (2) real-time learning loops, and (3) modular hardware-software co-design. Signal processing at the π-node level is tailored to the physical embodiment—allowing for motion-specific or material-specific response strategies. Learning is handled through a minimal but effective reinforcement update rule, enabling nodes to adapt weights in response to performance signals over time. Importantly, this learning is localized: individual modules do not require centralized orchestration to evolve their behavior.

A central advantage of this decentralized model is energy efficiency. By distributing computation and minimizing the need for global communication, the system reduces latency and energy draw—key factors for edge devices and embedded systems. Additionally, the modularity of π-0.5 makes it hardware-agnostic, capable of interfacing with a variety of microcontrollers, sensors, and actuators.

Another technical innovation is the system’s support for tactile and kinesthetic feedback integration. π-0.5 is built to accommodate proprioceptive sensing, which enhances its capacity to maintain adaptive behavior in response to physical stress, deformation, or external forces—especially relevant for soft robotics and wearable interfaces.

Preliminary Results and Application Scenarios

Initial demonstrations of π-0.5 showcase its adaptability across a variety of scenarios. In a soft robotic gripper prototype, the inclusion of π-0.5 nodes enabled the system to self-correct grip force based on the texture and compliance of held objects—without relying on pre-programmed models or external computation. Compared to a traditional control loop, this approach yielded a 30% improvement in grip accuracy and a 25% reduction in power consumption under similar test conditions.

In wearable prototypes, π-0.5 allowed for localized adaptation to different body movements, achieving smoother haptic feedback and better energy regulation during continuous use. These results highlight π-0.5’s potential not just in robotics but in augmentative human-machine interfaces, where context-sensitive responsiveness is critical.

Conclusion

π-0.5 marks a deliberate step away from monolithic AI architectures toward systems that closely couple intelligence with physical interaction. Rather than pursuing ever-larger centralized models, Physical Intelligence proposes a distributed, embodied approach grounded in modular design and real-time adaptation. This direction aligns with long-standing goals in cybernetics and biologically inspired computing—treating intelligence not as a product of abstraction, but as a property that emerges from constant physical engagement.

As AI continues to move into real-world systems, from wearables to autonomous machines, the need for low-power, adaptive, and resilient architectures will grow. π-0.5 offers a compelling foundation for meeting these requirements, contributing to a more integrated and physically grounded conception of intelligent systems.

Check out the Technical details. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. Don’t Forget to join our 90k+ ML SubReddit.

[Register Now] miniCON Virtual Conference on AGENTIC AI: FREE REGISTRATION + Certificate of Attendance + 4 Hour Short Event (May 21, 9 am- 1 pm PST) + Hands on Workshop

The post Researchers at Physical Intelligence Introduce π-0.5: A New AI Framework for Real-Time Adaptive Intelligence in Physical Systems appeared first on MarkTechPost.