Enhanced Performance with Large Datasets

User Problem

Users of Plotly Studio are currently limited in the size and performance of the datasets they can analyze within their applications. The current 200MB cap and reliance on Python's Pandas library for all data querying and manipulation lead to a sluggish experience. For instance, applying a new filter to the data causes the entire application to freeze or show loading bars for a significant duration, similar to the initial load time. Users are requesting the capability to work with datasets that are an order of magnitude larger while maintaining a responsive and fast application experience.

What is it?

This feature involves a foundational re-architecture of the data handling and querying engine within Plotly Studio applications. This shift would replace the current Pandas-based, in-memory processing with a high-performance database technology like DuckDB or a similar optimized solution.

The goal is to achieve significantly higher data throughput and faster query execution, ideally gaining performance as a side effect of switching to the new technology stack. This migration is the first step toward eventually backing applications with a native SQL layer rather than Python, which will enable future features to push query execution directly down to the original connected database.

What does it allow users to do?

This performance upgrade will allow users to:

• Work with much larger datasets: Users will be able to load and analyze datasets that are significantly larger than the current 200MB limit, opening up the analysis of high-volume business data.

• Experience faster, more responsive interactions: Core operations like applying global filters, aggregating data, or performing calculations will be nearly instantaneous, eliminating the frustrating loading times and application freezes that currently occur with moderately sized data.

• Build more complex, performant applications: Users can design more sophisticated, data-intensive applications without compromising on speed or user experience, leading to richer insights and more powerful data dashboards.

• Scale their data analysis: The ability to handle larger data volumes efficiently provides a clearer path for users to scale their analytical workflows as their data needs grow.