Basic Operations and Aggregations with Pandas in 2026: Vectorized Data Processing at Scale

Pandas basic operations — means, sorting, filtering, arithmetic — are the primitives every data pipeline relies on. In Pandas 2.x, these operations run on Apache Arrow-backed arrays by default, delivering 10-100x speedups over the row-by-row patterns most tutorials still teach.

This guide covers the complete operations stack: vectorized aggregations, Split-Apply-Combine groupby patterns, string accessors, and the .query() engine — all written for Pandas 2.2+ with PyArrow dtypes.

1. Vectorized Descriptive Statistics

Descriptive statistics in Pandas 2.x operate on contiguous Arrow arrays, not Python objects. This eliminates the boxing overhead that made describe() slow on large DataFrames:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'revenue': np.random.uniform(100, 50000, 1_000_000),
    'cost': np.random.uniform(50, 30000, 1_000_000),
}, dtype='float64[pyarrow]')

# Single-pass statistics — C-speed, zero Python loops
stats = df.describe()
print(stats.loc[['mean', 'std', '50%']])

# Direct aggregation — PyArrow-backed computation
mean_val = df['revenue'].mean()
std_val = df['revenue'].std()
median_val = df['revenue'].median()

# CoW-safe: assignments do not copy
df['profit'] = df['revenue'] - df['cost']

Why this matters: With dtype_backend="pyarrow", .mean() and .std() run on contiguous 64-bit buffers instead of boxed Python floats — 10-50x faster on million-row DataFrames.

2. Sorting with Index Preservation

Sorting in Pandas 2.x supports multi-column sorts with stable algorithms and preserves index semantics:

# Single-column descending sort
sorted_df = df.sort_values('revenue', ascending=False, kind='stable')

# Multi-column sort — primary by revenue, secondary by cost
sorted_df = df.sort_values(['revenue', 'cost'], ascending=[False, True])

# Sort by index — required before .loc[] range lookups
df_sorted = df.sort_index()

Stability matters: kind='stable' (Timsort) preserves row order for equal keys. Critical when sorting twice on different columns — unstable sorts scramble tied rows unpredictably.

3. Boolean Filtering and Masking

Boolean masking delegates entirely to Arrow bitmaps — no index traversal, no Python iteration:

# Compound boolean mask — vectorized C-speed evaluation
high_margin = df[(df['revenue'] > 10000) & (df['cost'] < 5000)]

# isin for set membership — hash-based lookup
categories = df[df['region'].isin(['NA', 'EU', 'APAC'])]

# String masks with Arrow-backed .str accessor
df['description'] = pd.array(['premium', 'standard', 'premium'], dtype='string[pyarrow]')
premium = df[df['description'].str.contains('premium', na=False)]

PyArrow acceleration: String masks on string[pyarrow] columns operate on contiguous null-bitmaps rather than Python objects — 3-10x faster with 50-70% less memory.

4. Vectorized Arithmetic and NumPy Integration

Pandas 2.x arithmetic operations dispatch directly to NumPy ufuncs on Arrow arrays:

# Element-wise arithmetic — no row iteration
df['margin_pct'] = (df['revenue'] - df['cost']) / df['revenue'] * 100

# NumPy ufunc integration — log, exp, sqrt on arrays
df['log_revenue'] = np.log1p(df['revenue'])
df['z_score'] = (df['revenue'] - df['revenue'].mean()) / df['revenue'].std()

# Clip — bounded clamping without apply
df['capped'] = df['revenue'].clip(lower=0, upper=100000)

Key change in 2.x: With Copy-on-Write enabled by default, arithmetic operations return new arrays — they never mutate the source DataFrame implicitly.

5. GroupBy Aggregations: Split-Apply-Combine

The Split-Apply-Combine pattern is Pandas' most powerful aggregation primitive. Pandas 2.x optimizes each phase with Cython kernels:

# Split-Apply-Combine — groupby aggregation
summary = df.groupby('region').agg(
    total_revenue=('revenue', 'sum'),
    avg_profit=('profit', 'mean'),
    max_cost=('cost', 'max'),
    order_count=('revenue', 'count')
).reset_index()

# Transform — broadcast group results back to rows
df['region_avg'] = df.groupby('region')['revenue'].transform('mean')
df['pct_of_region'] = df['revenue'] / df['region_avg'] * 100

Performance: groupby().agg() with named aggregations runs 2-5x faster than .apply(lambda) because the aggregation table is compiled to Cython — no Python function calls per group.

6. The .query() Method: Compiled Filtering

.query() compiles filter expressions via numexpr, which parallelizes across CPU cores:

# Basic query syntax — numexpr-optimized
result = df.query('revenue > 10000 and cost < 5000')

# Using local variables with @
threshold = 25000
result = df.query('revenue > @threshold')

# Arithmetic expressions inside query
result = df.query('(revenue / cost) > 5 and region in ["NA", "EU"]')

# String methods inside query (Pandas 2.x)
result = df.query('description.str.contains("premium")')

When to use: For DataFrames exceeding 100K rows, .query() outperforms equivalent boolean indexing by 2-5x because numexpr avoids creating intermediate temporary arrays.

7. String Accessor Operations

The .str accessor in Pandas 2.x leverages PyArrow's string kernels for zero-copy operations:

# Lowercasing — Arrow string kernel
df['email'] = df['email'].str.lower()

# Extraction — regex on Arrow string array
df['domain'] = df['email'].str.extract(r'@(.+\.\w+)')

# Concatenation — vectorized join
df['label'] = df['region'].str.cat(df['category'], sep='_')

Memory benefit: With string[pyarrow] dtype, .str operations modify contiguous buffers rather than creating new Python string objects per row.

8. Production Operations Patterns

1. Always use dtype_backend="pyarrow": Vectorized operations on Arrow arrays are 10-50x faster than object dtypes.
2. Prefer .agg() with named aggregations over .apply(): Compiled Cython kernels vs Python function calls.
3. Use .query() for complex filters on >100K rows: numexpr parallelizes across CPU cores.
4. Sort indices before .loc[] range lookups: Converts O(n) scans to O(log n) binary searches.
5. Avoid .apply(axis=1) for row-wise operations: Rewrite as vectorized expressions — 100-1000x faster.
6. Use .clip() instead of np.where for bounded values: Cleaner, faster, Arrow-native.
7. Cache groupby results: If referencing the same groupby multiple times, assign to a variable.

Vectorized operations are the core contract of Pandas 2.x — write data transformations as column-wide expressions, not row-by-row loops, and the engine does the rest.