Python数据处理5个必学技巧

从数据读取到清洗转换，这5个技巧覆盖了80%的日常数据处理场景。

一、技巧1：Pandas读取数据的5种方式

1.1 基础读取

import pandas as pddf = pd.read_csv('data.csv')df = pd.read_excel('data.xlsx', sheet_name='Sheet1')df = pd.read_json('data.json')from sqlalchemy import create_engineengine = create_engine('postgresql://user:pass@localhost/db')df = pd.read_sql('SELECT * FROM orders', engine)

1.2 高效读取大文件

df = pd.read_csv('large_data.csv', usecols=['name', 'age', 'city'])df = pd.read_csv('data.csv', dtype={    'age': 'int8',    'salary': 'float32',    'category': 'category'  # 低基数文本用category})chunk_size = 10000for chunk in pd.read_csv('huge_data.csv', chunksize=chunk_size):    process(chunk)  # 逐块处理

内存优化对比：

• 默认读取：占用内存 1.2GB
• 指定dtype后：占用内存 300MB（减少75%）

二、技巧2：链式调用写出优雅代码

Pandas支持链式调用，让代码更流畅：

df = pd.read_csv('orders.csv')df = df[df['amount'] > 100]           # 筛选金额>100df = df[df['status'] == 'completed']  # 筛选已完成df = df.sort_values('amount', ascending=False)  # 按金额降序df = df.head(10)                       # 取前10top_orders = (    pd.read_csv('orders.csv')    .query('amount > 100 and status == "completed"')    .sort_values('amount', ascending=False)    .head(10))result = (    pd.read_csv('sales.csv')    .assign(        month=lambda x: pd.to_datetime(x['date']).dt.month,        tax=lambda x: x['amount'] * 0.13    )    .groupby(['city', 'month'])    .agg(        total_amount=('amount', 'sum'),        order_count=('order_id', 'count'),        avg_amount=('amount', 'mean')    )    .reset_index()    .sort_values('total_amount', ascending=False))

链式调用三原则：

1. 每个方法返回新的DataFrame（不修改原数据）
2. 使用assign()添加新列
3. 使用query()筛选条件

三、技巧3：apply()的正确使用姿势

3.1 向量化操作优先

df['price_with_tax'] = df['price'].apply(lambda x: x * 1.13)df['price_with_tax'] = df['price'] * 1.13df['level'] = ''df.loc[df['score'] >= 90, 'level'] = '优秀'df.loc[(df['score'] >= 60) & (df['score'] < 90), 'level'] = '良好'df.loc[df['score'] < 60, 'level'] = '不及格'import numpy as npconditions = [    df['score'] >= 90,    df['score'] >= 60,    df['score'] < 60]choices = ['优秀', '良好', '不及格']df['level'] = np.select(conditions, choices, default='未知')

3.2 复杂逻辑才用apply

def calculate_discount(row):    if row['vip_level'] == 'gold' and row['amount'] > 1000:        return 0.2    elif row['amount'] > 500:        return 0.1    else:        return 0df['discount'] = df.apply(calculate_discount, axis=1)conditions = [    (df['vip_level'] == 'gold') & (df['amount'] > 1000),    df['amount'] > 500]choices = [0.2, 0.1]df['discount'] = np.select(conditions, choices, default=0)

性能对比：

• apply：处理100万行需8秒
• 向量化：处理100万行需0.05秒（快160倍）

四、技巧4：groupby聚合的5种常用模式

4.1 基础分组统计

sales = pd.DataFrame({    'city': ['北京', '上海', '北京', '深圳', '上海', '北京'],    'product': ['A', 'B', 'A', 'A', 'B', 'C'],    'amount': [100, 200, 150, 300, 250, 120],    'quantity': [10, 20, 15, 30, 25, 12]})city_sales = sales.groupby('city')['amount'].sum()print(city_sales)city_product = sales.groupby(['city', 'product'])['amount'].sum()print(city_product)

4.2 多指标聚合

agg_result = sales.groupby('city').agg({    'amount': ['sum', 'mean', 'max', 'min'],    'quantity': ['sum', 'mean', 'count']})print(agg_result)agg_result = sales.groupby('city').agg(    total_amount=('amount', 'sum'),    avg_amount=('amount', 'mean'),    max_amount=('amount', 'max'),    total_quantity=('quantity', 'sum'),    order_count=('amount', 'count'))

4.3 分组后过滤

result = sales.groupby('city').filter(lambda x: len(x) >= 2)print(result)result = sales.groupby('city').filter(lambda x: x['amount'].sum() > 400)

4.4 分组后转换

sales['city_pct'] = sales.groupby('city')['amount'].transform(lambda x: x / x.sum() * 100)print(sales)

***

五、技巧5：数据清洗的4个关键操作

5.1 处理缺失值

df = pd.DataFrame({    'name': ['张三', '李四', None, '王五'],    'age': [25, None, 30, 35],    'salary': [5000, 6000, 7000, None]})print(df.isnull().sum())df_drop = df.dropna()  # 删除任何含有NA的行df_drop = df.dropna(subset=['name'])  # 只删除name为NA的行df['age'] = df['age'].fillna(df['age'].mean())  # 均值填充df['salary'] = df['salary'].fillna(0)  # 固定值填充df['name'] = df['name'].fillna(method='ffill')  # 前向填充

5.2 去重

df = pd.DataFrame({    'id': [1, 2, 2, 3, 3, 4],    'name': ['张三', '李四', '李四', '王五', '王五', '赵六']})df_unique = df.drop_duplicates()df_unique = df.drop_duplicates(subset=['id'], keep='first')  # 保留第一个df_unique = df.drop_duplicates(keep=False)  # 删除所有重复项

5.3 数据类型转换

df = pd.DataFrame({    'date': ['2024-01-01', '2024-01-02', '2024-01-03'],    'price': ['19.99', '29.99', '39.99'],    'quantity': ['100', '200', '300']})df['date'] = pd.to_datetime(df['date'])df['price'] = pd.to_numeric(df['price'])df['quantity'] = df['quantity'].astype(int)df['category'] = df['category'].astype('category')

5.4 字符串处理

df = pd.DataFrame({    'name': ['张三', '李四', '王五'],    'phone': ['13800138000', '13900139000', '13700137000']})df['name_upper'] = df['name'].str.upper()  # 大写df['phone_masked'] = df['phone'].str.replace(r'\d{7}', '*******', regex=True)  # 脱敏df['name_length'] = df['name'].str.len()  # 长度df['area_code'] = df['phone'].str[:3]  # 取前3位

***

六、NumPy高效数组操作

6.1 广播机制

import numpy as nparr = np.array([1, 2, 3, 4, 5])print(arr + 10)     # [11, 12, 13, 14, 15]print(arr * 2)      # [2, 4, 6, 8, 10]a = np.array([[1, 2], [3, 4]])b = np.array([10, 20])print(a + b)  # [[11, 22], [13, 24]] (b自动扩展到a的形状)

6.2 布尔索引

arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])result = arr[arr > 5]print(result)  # [6, 7, 8, 9, 10]mask = (arr > 3) & (arr < 8)print(arr[mask])  # [4, 5, 6, 7]arr[arr > 5] = 0print(arr)  # [1, 2, 3, 4, 5, 0, 0, 0, 0, 0]

6.3 聚合与统计

arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])print(arr.sum())         # 45 (全部求和)print(arr.sum(axis=0))   # [12 15 18] (按列求和)print(arr.sum(axis=1))   # [6 15 24] (按行求和)print(arr.mean())        # 5.0print(arr.std())         # 2.58print(arr.max(axis=1))   # [3 6 9]

七、总结