Brief

Researchers have introduced TAPEX, a novel pre-training method for enhancing table understanding in language models. This approach leverages a "table-to-text" objective, allowing models to generate textual representations of tabular data. TAPEX demonstrates improved performance on various table-related downstream tasks, offering a more efficient way to train models on structured information without requiring extensive real-world datasets. AI

OpenAI is expanding access to its DALL-E 2 research preview, inviting up to 1,000 new users weekly from its waitlist. The company has focused on enhancing safety systems, with less than 0.05% of shared images flagged for policy violations. OpenAI is also actively working to address biases in the model inherited from its training data, requesting early users to avoid sharing photorealistic images with faces. AI

BigScience is currently training a large language model, attracting significant global attention. Concurrently, NVIDIA has unveiled its newest generation of GPUs, the "Hopper" series. These developments, alongside other AI-related news, were discussed in a recent episode of Practical AI. AI

Hugging Face has published a guide detailing how to fine-tune a semantic segmentation model using a custom dataset. The tutorial focuses on the SegFormer model, demonstrating the process of adapting it for specific segmentation tasks. This guide is intended to help users leverage pre-trained models and tailor them to their unique data requirements. AI

OpenAI has introduced new GPT-3 and Codex capabilities that allow for editing and inserting content within existing text, moving beyond simple text completion. The 'insert' feature enables contextually relevant additions in the middle of text or code, improving applications like long-form writing and code generation. Additionally, a new 'edits' endpoint allows for modifications to existing text based on specific instructions, useful for tasks such as refactoring code, changing tone, or fixing errors. These features are now available in beta via the OpenAI API and are being piloted in tools like GitHub Copilot. AI

Hugging Face has introduced two new text generation techniques for its Transformers library: contrastive search and constrained beam search. Contrastive search aims to produce more human-like text by balancing likelihood and distinctiveness, while constrained beam search allows users to guide the generation process with specific rules or patterns. These methods offer developers more control and improved quality for text generation tasks within the Hugging Face ecosystem. AI

Researchers have developed an AI system capable of quickly predicting protein attachments, a significant advancement in biological research. Additionally, a new self-supervised algorithm from Meta AI demonstrates high performance across speech, vision, and text modalities. DeepMind has also announced an AI coding engine that matches the proficiency of an average human programmer. AI

Hugging Face has released a guide on fine-tuning the Vision Transformer (ViT) model for image classification tasks. The tutorial utilizes the 🤗 Transformers library, demonstrating how to adapt a pre-trained ViT model to a specific dataset. This process allows developers to leverage powerful pre-trained models for custom image recognition applications without training from scratch. AI

EleutherAI has released GPT-NeoX-20B, a 20 billion parameter open-source language model trained using their GPT-NeoX framework. This model is notable for being the largest publicly accessible pretrained autoregressive language model to date. The release aims to facilitate research into the safe use of AI systems, with the model available via inference services and a public release scheduled after a seven-day delay. AI

OpenAI has developed a neural theorem prover for the Lean formal proof assistant that can solve challenging high-school olympiad math problems. The system utilizes a language model to discover proofs, iteratively improving its performance by using newly found proofs as training data. This approach achieved a new state-of-the-art on the miniF2F benchmark, outperforming previous methods. AI

Hugging Face has released Infinity, a new inference engine designed to optimize large language model performance on modern CPUs. This engine achieves millisecond latency by leveraging techniques like quantization and efficient memory management. The goal is to make powerful LLMs more accessible and cost-effective for a wider range of applications without requiring specialized hardware. AI

Hugging Face has released updates to its Transformers library, enhancing the Wav2Vec2 model for automatic speech recognition (ASR). The library now supports processing large audio files by implementing chunking, which breaks down large files into smaller, manageable segments. Additionally, performance is boosted through the integration of n-grams, further improving the accuracy and efficiency of speech recognition tasks. AI

Perceiver IO is a new AI model architecture developed by DeepMind that utilizes a fully attentional mechanism to process information from various modalities. Unlike previous models that required modality-specific input processing, Perceiver IO can handle diverse data types like images, audio, and text directly. This approach aims to create a more scalable and unified framework for multimodal AI research and applications. AI

Hugging Face has released CodeParrot, a new large language model specifically trained for code generation. The model was built from scratch using a novel training approach that emphasizes efficiency and performance. CodeParrot is designed to assist developers by generating code snippets, completing code, and potentially aiding in debugging tasks. AI

Hugging Face has released Snowball Fight, a new machine learning environment designed for training agents. This environment is built using the ML-Agents toolkit and aims to provide a platform for developing and testing AI agents in a simulated setting. The release is intended to foster innovation in reinforcement learning and agent-based AI development within the community. AI

METR, an AI safety research organization, detailed its 2023 accomplishments, including developing methodologies for evaluating AI agents on autonomous tasks and contributing to OpenAI's GPT-4 system card. The organization also proposed "Responsible Scaling Policies" (RSPs), a framework for AI safety that gained traction among researchers and companies like Anthropic and OpenAI. Additionally, METR partnered with the UK AI Safety Institute and evaluated GPT-5.1 for catastrophic risks. AI

The BigScience research workshop, a year-long initiative by Hugging Face, has released the T0 family of AI models. These models are specifically designed to explore zero-shot multitask learning in natural language processing. The T0 models demonstrate the potential for AI to generalize across various tasks without explicit training for each one. AI

Hugging Face has partnered with Intel to optimize PyTorch distributed fine-tuning using Intel's latest technologies. This collaboration focuses on enhancing performance and efficiency for large language model training. The integration aims to leverage Intel's hardware advancements to accelerate the fine-tuning process, making it more accessible and faster for researchers and developers. AI

OpenAI has developed a new system capable of solving grade school math word problems with nearly double the accuracy of previous GPT-3 models. This system achieves approximately 90% of the performance of real children in the 9-12 age range by training the model to recognize and correct its own errors through repeated attempts. The approach involves using verifiers to evaluate multiple candidate solutions, selecting the best one, which offers a significant performance boost and appears to scale more effectively with data than simply increasing model size. AI

Hugging Face has announced a new initiative, "Machine Learning as Code," aiming to standardize how machine learning models are developed, shared, and deployed. This approach treats ML models like software code, emphasizing version control, reproducibility, and collaboration. The goal is to streamline the ML lifecycle, making it more accessible and efficient for developers and researchers. AI

Hugging Face has released a guide on fine-tuning the CLIP model using remote sensing images and their corresponding captions. This process involves adapting the pre-trained CLIP model to better understand and associate visual information from satellite imagery with textual descriptions. The guide details the steps and considerations for this specialized application of CLIP, enabling more accurate analysis and retrieval of geospatial data. AI

Hugging Face is hosting a series of events and releasing new features throughout the summer. These initiatives aim to foster community engagement and advance the open-source AI ecosystem. Key highlights include new model releases, educational content, and opportunities for developers to collaborate and showcase their work. AI

Hugging Face has introduced a new framework enabling collaborative training of large language models over the internet. This approach allows multiple parties to contribute to training without sharing their raw data, addressing privacy and security concerns. The system leverages techniques to ensure that individual data remains private while still enabling the collective model to learn from diverse datasets. AI

EleutherAI has released a retrospective detailing their work over the past year and a half. Key achievements include the development of the open-source LLM GPT-NeoX-20B and contributions to text-to-image generation models like VQGAN-CLIP. The organization has also seen several members depart to found new AI research entities focused on alignment, preference learning, and biomedical applications. AI

Hugging Face has introduced SetFit, a novel few-shot learning approach that achieves state-of-the-art performance without requiring prompt engineering. This method utilizes a two-stage process: first, it fine-tunes a model on a small set of labeled data, and then it generates synthetic data from this fine-tuned model to further train it. SetFit has demonstrated impressive results, outperforming prompt-based methods like few-shot GPT-3 on several benchmarks, and is available as an open-source library. AI

EleutherAI has detailed its reasoning for releasing large language models, emphasizing that open access is crucial for advancing AI safety research. The organization argues that significant safety studies, particularly in model interpretability, can only be effectively conducted with access to these powerful models. They believe that the potential dangers of current large language models are not world-ending and that releasing them allows for critical safety research to be performed before models become significantly more powerful and potentially uncontrollable. Furthermore, EleutherAI contends that attempts to restrict access to this technology are futile, as well-funded actors can replicate it, making open release the best strategy to empower society to study and utilize it for beneficial purposes. AI

EleutherAI has estimated the parameter counts of OpenAI's API models by comparing their performance on various tasks to known benchmarks. Their analysis suggests that models like Ada, Babbage, Curie, and Davinci correspond to approximately 350 million, 1.3 billion, 6.7 billion, and 175 billion parameters, respectively. While not official figures, these estimates provide a strong indication of the scale of OpenAI's deployed models. AI

Researchers at EleutherAI investigated how different few-shot description prompts affect GPT-3's performance on the SST benchmark. Their experiments revealed that smaller GPT-2 models performed poorly and inconsistently, with performance not strictly increasing with model size. Surprisingly, the study found no correlation between different GPT models regarding which prompts yielded the best results, challenging the expectation that similar models would favor similar prompting strategies. AI

Researchers at EleutherAI explored the impact of fine-tuning the GPT-Neo 2.7B model on a diverse set of downstream tasks. They observed that while the fine-tuned model did not universally outperform the base model, it showed significant improvements on certain tasks like ANLI. However, this specialization came at the cost of performance degradation on tasks not included in the fine-tuning set, such as LAMBADA and PubMedQA, indicating a potential for catastrophic forgetting. AI

PolyAI CEO Nikola Mrkšić discussed advancements in conversational AI and the development of next-generation voice assistants capable of human-level conversations. The company's ConveRT model has demonstrated superior performance compared to BERT and GPT-based models in evaluations, particularly in understanding various languages and accents. PolyAI's technology aims to enhance customer service interactions through more sophisticated voice assistant capabilities. AI

BigBird is a novel attention mechanism designed to address the quadratic complexity of standard Transformer models. It achieves this by employing a sparse attention pattern, which includes global, window, and random attention, allowing it to process significantly longer sequences than traditional Transformers. This innovation makes BigBird particularly effective for tasks requiring long-range dependencies, such as document summarization and question answering on extensive texts. AI

Hugging Face has released a series of blog posts detailing how to fine-tune various Wav2Vec2 and Whisper models for Automatic Speech Recognition (ASR) tasks using their Transformers library. These guides cover adapting models for low-resource scenarios, multilingual applications, and specific languages like English. The tutorials emphasize practical implementation for researchers and developers working with speech data. AI

This blog post from Hugging Face discusses the advancements in long-range Transformers, a type of neural network architecture. It explores how these models are being developed to handle longer sequences of text, overcoming previous limitations. The post likely delves into the technical aspects and potential applications of these more capable Transformer models. AI

OpenAI researchers have identified "multimodal neurons" within their CLIP model, which respond to concepts regardless of whether they are presented visually, symbolically, or textually. This discovery offers insight into how CLIP achieves high accuracy on challenging datasets by abstracting concepts, similar to how neurons in the human brain function. The findings suggest a common mechanism for abstraction in both artificial and natural vision systems, potentially explaining model versatility and compactness. AI

The Machine Learning Compilation (MLC) group, led by Tianqi Chen at CMU, is developing frameworks like MLC Chat and Web LLM to enable running large language models on consumer hardware, including iPhones and web browsers. This initiative aims to mitigate the current GPU shortage by allowing models to run locally on devices with AMD cards or even just CPUs. Projects like Hugging Face's text-to-webapp generator and Gradio are also contributing to easier deployment and accessibility of ML models for developers and end-users. AI

Hugging Face has integrated ZeRO (Zero Redundancy Optimizer) into its libraries, leveraging DeepSpeed and FairScale. This enhancement allows for more efficient training of large language models by reducing memory redundancy across distributed training setups. The optimization enables fitting larger models into memory and accelerating the training process. AI

A new paper explores the critical role of user state representation in contextual multi-armed bandit (CMAB) recommender systems, finding that variations in state representation can yield greater performance improvements than changes to the bandit algorithm itself. The research highlights that no single embedding or aggregation strategy is universally superior, emphasizing the need for domain-specific evaluations. Another study introduces BEAR, a novel fine-tuning objective for Large Language Models (LLMs) in recommendation tasks that explicitly accounts for beam search behavior during training to address inconsistencies between training and inference. Additionally, a paper proposes a methodology to measure the stability and plasticity of recommender systems, evaluating how models adapt to retraining and changes in data patterns. AI

OpenAI has introduced CLIP, a neural network designed to learn visual concepts from natural language supervision. This model can perform a wide range of image classification tasks without specific training for each benchmark, leveraging the vast amount of text paired with images available online. CLIP aims to overcome limitations of traditional computer vision models, such as the cost of creating datasets and the narrow focus of task-specific training, by achieving robust performance across various benchmarks with zero-shot capabilities. AI

OpenAI has detailed a new method for generating images from text using CLIP latents, employing a two-stage process with a prior and a decoder. This approach enhances image diversity while maintaining photorealism and caption similarity, and allows for language-guided image manipulations. Separately, OpenAI also introduced DALL-E, a 12-billion parameter GPT-3 variant capable of creating images from text descriptions, demonstrating abilities like combining concepts and rendering text. AI