7 Python案例

7.2 拆分數據

7.2.1 解析數據庫字段里的數據

需求：從數據庫字段里解析得到結構化數據。

解決方法：通過Python的pandas以及內置的函數完成該需求。

?

?Python代碼：

?

import numpy as np import pandas as pd import os from sqlalchemy import create_engine import pymysql conn = pymysql.connect(host='192.168.56.32',port = 3306,user='test',passwd='qwert@765',db ='mydb',charset='utf8' )def tidy_split(df, column, sep='|', keep=False):"""Split the values of a column and expand so the new DataFrame has one splitvalue per row. Filters rows where the column is missing.Params------df : pandas.DataFramedataframe with the column to split and expandcolumn : strthe column to split and expandsep : strthe string used to split the column's valueskeep : boolwhether to retain the presplit value as it's own rowReturns-------pandas.DataFrameReturns a dataframe with the same columns as `df`."""indexes = list()new_values = list()df = df.dropna(subset=[column])for i, presplit in enumerate(df[column].astype(str)):values = presplit.split(sep)if keep and len(values) > 1:indexes.append(i)new_values.append(presplit)for value in values:indexes.append(i)new_values.append(value)new_df = df.iloc[indexes, :].copy()new_df[column] = new_valuesreturn new_dfdef read_table(cur, sql_order):try:cur.execute(sql_order)??data = cur.fetchall()frame = pd.DataFrame(list(data))except:? # , e:frame = pd.DataFrame()# print e# continuereturn framedef splitRFIDLog(cur, sql_order):rfidLogtab=read_table(cur, sql_order)rfidLogtab.columns = ['skuid', 'rfid','deviceid','records']### 設置列的最大長度pd.set_option('max_colwidth', 1000)newtab1=tidy_split(rfidLogtab.iloc[1:4,0:4], 'records', sep='<br >') ###僅取3行newtab1.columns=['skuid', 'rfid','deviceid','records']newtab1=newtab1[newtab1.records.str.contains("入庫") == False]newtab1['records'] = newtab1.records.str.replace(' 已售 ', '')newtab1['records'] = newtab1.records.str.replace(' 入柜 ', '')newtab2=tidy_split(newtab1,'records', sep='[')newtab3 = tidy_split(newtab2.iloc[1::2, :], 'records', sep='-')newtab4 = tidy_split(newtab3, 'records', sep=']')newtab4 = pd.DataFrame(newtab4.records.str.replace('deviceid=', ''))newtab5=newtab4.records.str.slice(0,21)newtab5=pd.DataFrame(newtab5,columns=['records'])print(newtab5)df = pd.DataFrame(newtab5.records.values.reshape(-1, 3),columns=['prestatus', 'endstatus','deviceid'])df = df.reset_index(drop=True)print(df)timedf=newtab2[newtab2.iloc[0::1, :].records.str.contains("deviceid") == False]#.iloc[0::1, :]timedf = timedf.reset_index(drop=True)splitRFIDLogTab=pd.concat([timedf,df],axis=1, join='inner')#將拆分的RFID的數據分到CSV中splitRFIDLogTab.to_csv('E:\\a.csv', sep=',', header=True, index=False)conn.commit()conn.close() if __name__ == '__main__':cur = conn.cursor()sql_order = "SELECT skuid, rfid, deviceid, records FROM main_log limit 20;"splitRFIDLog(cur,sql_order)

將生成的結果寫入到csv中，見如下示例結果：

skuid	rfid	deviceid	records	prestatus	endstatus	deviceid
1	5133C90F30DCC1EE00005133C90F30DCC1EE0000	898602c9981730091839	2017/11/2 10:52	3	4	898602c9981730091839

總結

以上是生活随笔為你收集整理的Python数据库字段拆分数据的全部內容，希望文章能夠幫你解決所遇到的問題。

如果覺得生活随笔網站內容還不錯，歡迎將生活随笔推薦給好友。