Apache Parquet
PulseAugur coverage of Apache Parquet — every cluster mentioning Apache Parquet across labs, papers, and developer communities, ranked by signal.
4 day(s) with sentiment data
-
Databricks introduces native FILE type for multimodal data in lakehouse
Databricks has introduced a new beta feature called FILE type, designed to store unstructured data like documents, images, and videos as a native column within tables. This innovation aims to integrate multimodal data d…
-
New Lakehouse Method Enhances Vector Search with File Pruning
Researchers have developed a novel method for integrating approximate nearest-neighbor (ANN) search with structured data filtering within an open lakehouse table format, specifically Apache Iceberg over Parquet. This ap…
-
AI applications explored: security analysis, development, and documentation
Several articles on Qiita discuss the application of AI in various technical contexts. One article details the observation and AI-driven analysis of attacks exploiting SOCKS proxies. Another explores a "sloppy AI-driven…
-
Microsoft Fabric Warehouses introduce "time travel" for data recovery
Microsoft Fabric Warehouses now offer a "time travel" feature, allowing users to query data as it existed at a specific past point in time. This capability is enabled by the underlying Delta storage architecture, which …
-
Databricks makes Variant data type generally available for faster data ingestion
Databricks has announced the general availability of its Variant data type, designed to improve the ingestion and querying of semi-structured data. This feature, along with Variant Shredding, uses predictive optimizatio…
-
AI Teams Streamline Data Pipelines with DuckDB, Polars, and Parquet
This article details how AI teams are adopting a more efficient data pipeline architecture. It highlights the use of DuckDB, Polars, and Apache Parquet as key components for building faster and simpler data processing s…
-
Apache Arrow and Parquet: Columnar Storage Explained for Data Engineers
This article uses a fictional dialogue between two data engineers, Joey and Chandler, to explain the benefits of Apache Arrow and Apache Parquet for large-scale data processing. Chandler explains that row-based storage …
-
Apache Spark Catalyst Optimizer: Internals and Performance Pitfalls
This article delves into the internal workings of the Apache Spark Catalyst optimizer, explaining how SQL queries and DataFrame calls are transformed into efficient physical execution plans. It highlights a real-world i…
-
Apache Spark query decomposition leads to costly data recomputation
A common pitfall in Apache Spark involves accidental recomputation of data due to lazy evaluation and the way Spark decomposes queries. A single `df.count()` operation, when placed before an action like `.write()` and w…
-
AI tech stacks, data analysis, and multimodal embeddings explored
This cluster of articles from Qiita, shared on Mastodon, explores various AI applications and technical stacks. One article discusses using TypeScript as a primary language for AI code generation, supplemented by Python…
-
Databricks launches OpenSharing for AI agents, models, and data
Databricks has introduced OpenSharing, an evolution of its Delta Sharing protocol designed for the agentic AI era. This new open-source protocol, now hosted by the Linux Foundation, expands beyond data sharing to encomp…
-
MLSkip improves database filtering with lightweight metadata
Researchers have developed MLSkip, a novel technique to improve data skipping for machine learning filters in databases. Traditional methods are ineffective with costly, black-box ML models used in filter predicates. ML…
-
Java parser Hardwood boosts Apache Parquet speed with AI
Gunnar Morling, known for the 1 Billion Row Challenge, has released Hardwood, a new Java parser for Apache Parquet. This parser is designed to be zero-dependency and ultra-fast, utilizing page-level parallelization with…
-
AWS integrates custom Graviton chips into Redshift analytics stack
Amazon Web Services has launched new Graviton-powered RG instances for its Redshift data warehousing service. These instances integrate data warehouse and data lake analytics, aiming to improve performance and reduce co…
-
OptimusKG: New multimodal biomedical graph unifies diverse life science data
Researchers have developed OptimusKG, a novel multimodal knowledge graph designed to unify diverse biomedical data. This graph integrates structured and semi-structured resources, encompassing molecular, anatomical, cli…