SQL-backed Apps

User Problem

Currently, Plotly Studio apps rely on in-memory processing (Pandas) for queries, which can lead to slow performance and scalability issues, especially when handling large datasets or complex analytical operations. Furthermore, the existing architecture requires custom or limited methods for connecting to diverse data sources, complicating the process for users who need to integrate their applications with existing database infrastructure.

What Is It?

This feature is a foundational architectural enhancement that transitions Plotly Studio apps to be SQL-backed, initially leveraging a high-performance, in-process SQL database technology like DuckDB. This change means that data within the app is treated as an SQL database, and all subsequent in-app queries, filtering, and aggregation operations are executed using SQL.

What Does It Allow Users to Do?

This enhancement offers several significant benefits:

• Significantly Increased Query Performance: By utilizing a robust SQL engine, the feature dramatically accelerates the speed at which in-app queries, filters, and aggregations are executed, leading to a much smoother and more responsive user experience, particularly with larger datasets. It also allows the resulting application to go beyond the current 10k row limitation in the table visualization.

• Robust Built-in Connection Options: The underlying SQL engine immediately provides users with standardized and reliable built-in connection methods to load data from various file types (like CSV, Parquet, and JSON) and other common data sources.

• Future-Proofing for External Database Connectivity: This foundational transition sets the stage for a longer-term capability where Plotly Studio apps could be enabled to directly push-down queries to separate, external SQL databases (e.g., PostgreSQL, Snowflake, BigQuery). This would allow users to leverage the processing power of their existing external data warehouses without needing to ingest the entire dataset into the app first.