python制作mysql数据迁移脚本


Posted in Python onJanuary 01, 2019

用python写了个数据迁移脚本,主要是利用从库将大的静态表导出表空间,载导入到目标实例中。

#!/usr/bin/env python3
#-*- coding:utf8 -*-
#author:zhanbin.liu
#!!!!!DB必须同版本
#python3环境  pip3 install pymysql paramiko

import os
#from pathlib import Path
import sys
import pymysql
import paramiko

#每次只能迁移一个DB下的表,到指定DB
#GRANT SELECT, CREATE, RELOAD, ALTER, LOCK TABLES ON *.* TO 'data_migration'@'192.168.%' IDENTIFIED BY 'data_migration@123';
tables='sqlauto_cluster,sqlauto_user'    #以,分割的字符串,如a,b,c
tableList = tables.split(',')
sourceIp = '192.168.1.101'
sourceDataBase = '/data/mysql/3306/data'
sourceDbName = 'inception_web'
sourceDataDir = os.path.join(sourceDataBase,sourceDbName)
desIp = '192.168.1.102'
desDataBase = '/data/mysql/3306/data'
desDbName = 'inception_web'
desDataDir = os.path.join(desDataBase,desDbName)

# for table in tableList:
#   desFile = Path("%s/%s.ibd" %(desDataDir,table))
#   print(desFile)
#   if desFile.is_file():
#     print("ok")
#   else:
#     print("no")

comUser = 'data_migration'
comPwd = 'data_migration@123'
comPort = 3306

client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

def table_judge():
  print("table_judge")
  sourceTableExist = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  desTableExist = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  sourceTables = []
  desTables = []
  cursor_source = sourceTableExist.cursor()
  cursor_des = desTableExist.cursor()

  for table in tableList:
    #print(table)
    cursor_source.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (sourceDbName,table))
    sourceTable_tmp = cursor_source.fetchall()
    cursor_des.execute("select TABLE_NAME from information_schema.TABLES where TABLE_SCHEMA='%s' and TABLE_NAME='%s';" % (desDbName,table))
    desTable_tmp = cursor_des.fetchall()
    #print(desTable_tmp)
    if sourceTable_tmp is ():
      sourceTables.append(table)
    if desTable_tmp is not ():
      desTables.append(desTable_tmp[0][0])
  sourceTableExist.close()
  desTableExist.close()

  s=d=0
  if sourceTables != []:
    print('迁移源不存在将要迁移的表:',sourceIp,sourceDbName, sourceTables,' 请检查')
    s=1
  if desTables != []:
    print('目标库存在将要迁移的表:',desIp,desDbName,desTables,' 请移除')
    d=1
  if s == 1 or d == 1:
    sys.exit()

def data_sync():
  print('data_sync')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在同步表:",table)
    cursor_db_source.execute("show create table %s;" % (table))
    createTableSQL = cursor_db_source.fetchall()[0][1]
    print(createTableSQL)
    try:
      cursor_db_des.execute(createTableSQL)
    except Exception as error:
      print(error)
    cursor_db_source.execute("flush table %s with read lock;" % (table))
    cursor_db_des.execute("alter table %s discard tablespace;" % (table))

    client.connect(sourceIp, 22, 'root')
    stdin1, stdout1, stderr1 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".ibd", desIp, desDataDir))
    stdin2, stdout2, stderr2 = client.exec_command("scp %s %s:%s " % (sourceDataDir+"/"+table+".cfg", desIp, desDataDir))
    a_e_1 = stderr1.readlines()
    a_e_2 = stderr2.readlines()
    if a_e_1 != [] or a_e_2 != []:
      print(a_e_1,a_e_2)
      sys.exit()
    client.close()

    client.connect(desIp, 22, 'root')
    stdin3, stdout3, stderr3 = client.exec_command("chown -R mysql.mysql %s*" % (desDataDir+"/"+table))
    a_e_3 = stderr3.readlines()
    if a_e_3 != []:
      print(a_e_1, a_e_2)
      sys.exit()
    client.close()
    #cursor_db_source.execute("select sleep(10);")
    cursor_db_source.execute("unlock tables;")
    cursor_db_des.execute("alter table %s import tablespace;" % (table))
    print("同步完成")

  cursor_db_source.close()
  cursor_db_des.close()

def data_checksum():
  print('data_checksum')
  db_source = pymysql.connect(sourceIp,comUser,comPwd,sourceDbName,comPort,charset='utf8')
  db_des = pymysql.connect(desIp,comUser,comPwd,desDbName,comPort,charset='utf8')
  cursor_db_source = db_source.cursor()
  cursor_db_des = db_des.cursor()

  for table in tableList:
    print("正在校验表:", table)
    cursor_db_source.execute("checksum table %s;" % (table))
    ck_s = cursor_db_source.fetchall()[0][1]
    cursor_db_des.execute("checksum table %s;" % (table))
    ck_d = cursor_db_des.fetchall()[0][1]
    if ck_s != ck_d:
      print("表不一致:",table)
    else:
      print("表一致:",table)

  cursor_db_source.close()
  cursor_db_des.close()

if __name__ == "__main__":
  table_judge()
  data_sync()
  data_checksum()
  print('haha')
Python 相关文章推荐
Python3.x对JSON的一些操作示例
Sep 01 Python
python实现对excel进行数据剔除操作实例
Dec 07 Python
基于并发服务器几种实现方法(总结)
Dec 29 Python
python os用法总结
Jun 08 Python
解决pycharm无法识别本地site-packages的问题
Oct 13 Python
django框架实现模板中获取request 的各种信息示例
Jul 01 Python
在Pytorch中计算自己模型的FLOPs方式
Dec 30 Python
Python定义函数实现累计求和操作
May 03 Python
Python调用ffmpeg开源视频处理库,批量处理视频
Nov 16 Python
python实现scrapy爬虫每天定时抓取数据的示例代码
Jan 27 Python
pytorch 如何使用amp进行混合精度训练
May 24 Python
Python 实现绘制子图及子图刻度的变换等问题
May 31 Python
在python中将字符串转为json对象并取值的方法
Dec 31 #Python
对python中Json与object转化的方法详解
Dec 31 #Python
python使用zip将list转为json的方法
Dec 31 #Python
python 获取utc时间转化为本地时间的方法
Dec 31 #Python
python 实现UTC时间加减的方法
Dec 31 #Python
Python从单元素字典中获取key和value的实例
Dec 31 #Python
对Python 两大环境管理神器 pyenv 和 virtualenv详解
Dec 31 #Python
You might like
来自PHP.NET的入门教程
2006/10/09 PHP
Javascript select下拉框操作常用方法
2009/11/09 Javascript
读jQuery之七 判断点击了鼠标哪个键的代码
2011/06/21 Javascript
FF火狐下获取一个元素同类型的相邻元素实现代码
2012/12/15 Javascript
JavaScript设计模式之策略模式实例
2014/10/10 Javascript
js+css实现的圆角边框TAB选项卡滑动门代码分享(2款)
2015/08/26 Javascript
JavaScript知识点总结(五)之Javascript中两个等于号(==)和三个等于号(===)的区别
2016/05/31 Javascript
利用yarn实现一个webpack+react种子
2016/10/25 Javascript
详解VueJs前后端分离跨域问题
2017/05/24 Javascript
vuex与组件联合使用的方法
2018/05/10 Javascript
js canvas实现写字动画效果
2018/11/30 Javascript
微信小程序发送短信验证码完整实例
2019/01/07 Javascript
react使用CSS实现react动画功能示例
2020/05/18 Javascript
vue 函数调用加括号与不加括号的区别
2020/10/29 Javascript
[02:20]DOTA2亚洲邀请赛 IG战队出场宣传片
2015/02/07 DOTA
基于Django URL传参 FORM表单传数据 get post的用法实例
2018/05/28 Python
Python3使用SMTP发送带附件邮件
2020/06/16 Python
对python遍历文件夹中的所有jpg文件的实例详解
2018/12/08 Python
Python 做曲线拟合和求积分的方法
2018/12/29 Python
python爬虫神器Pyppeteer入门及使用
2019/07/13 Python
Python使用itchat模块实现简单的微信控制电脑功能示例
2019/08/26 Python
完美解决Pycharm中matplotlib画图中文乱码问题
2021/01/11 Python
使用HTML5 Canvas为图片填充颜色和纹理的教程
2016/03/21 HTML / CSS
美国礼品卡商城: Gift Card Mall
2017/08/25 全球购物
什么是命名空间(NameSpace)
2015/11/24 面试题
什么是动态端口(Dynamic Ports)?动态端口的范围是多少?
2014/12/12 面试题
业务总经理岗位职责
2014/02/03 职场文书
销售员个人求职的自我评价
2014/02/10 职场文书
股份转让协议书
2014/04/12 职场文书
演讲稿的写法
2014/05/19 职场文书
先进教育工作者事迹材料
2014/12/23 职场文书
使用Selenium实现微博爬虫(预登录、展开全文、翻页)
2021/04/13 Python
超外差式晶体管收音机的组装与统调
2021/04/22 无线电
浅谈如何保证Mysql主从一致
2022/03/13 MySQL
宝塔更新Python及Flask项目的部署
2022/04/11 Python
错误码NET::ERR_CERT_DATE_INVALID证书已过期解决方法?
2022/07/07 数码科技