Dashboard Migration Tool

Intention

This is a dashboard migration tool for DataTalk that allows importing and exporting dashboards between different environments. The Project allows a modular structure with a clear separation of concerns.

Problems

  • We use Helm to deploy private and public instances. However, due to network partition between instances. Data transmission over the network becomes impossible.
  • To achieve dashboard migration between instances, we need a way to export the data from one instance and import it to another instance.
  • Exporting from the Database is difficult due to
    • high database privilege demand and risk of messing up the database
    • unique key duplication and table definition mismatch
    • complex nested dependencies between records

Solutions

  • We use Open API to achieve data export and import. To consolidate problems with privilege, version mismatch, and complex dependency between metadata.
  • We used Python for the programming language. By using that, we are able to provide the dashboard migration tool with no code intrusion for the instance, higher version mismatch tolerance, and cleaner implementations for nested dependencies.
  • The tool boosted migration speed by 650%. Migrating by hand requires 2 person-days per dashboard, with the tool it is 0.2 person days with manual adjustment.

Implementation

Key Features

  • Interactive CLI interface
  • Automated batch processing
  • Secure password handling
  • Configuration-driven setup
  • Support for both manual and automated migrations
  • Comprehensive logging and error handling
  • Easy maintenance, testing, and extension of the tool
  • Keeping concerns separated and dependencies clear

Core Components

Command Line Interface

Provides the CLI interface and orchestrates the workflow

  • Interactive mode with prompts
  • Automated mode with --import-all and --export-all flags
  • Handles both import and export flows

Data Management

  • Configuration: Import/Export environment settings
  • File Handling: Handles for dump file (JSON) and middle file (CSV)
  • Secrets Handling: Handles encryption/decryption for sensitive data

Export Flow Components

  • List available dashboards
  • Export dashboard configurations
  • Export data sources
  • Export datasets and fields

Import Flow Components

  • Imports dashboard configurations
  • Handles dashboard creation
  • Creates data sources in target instance
  • Creates datasets with fields

Utility Components

Dashboard Parser

  • Parses dashboard configurations
  • Extracts data dependencies

 

Dashboard Replacer

  • Handles ID mapping between instances
  • Updates references in dashboard configs

 

Open API Client

  • Handles API communication
  • Manages authentication and requests

Workflow

Export Process

  1. List available dashboards
  2. Allow dashboard selection
  3. Export selected dashboards
  4. Extract data dependencies
  5. Export related data sources and datasets

Import Process

  1. Load exported configurations
  2. Import data sources first
  3. Import datasets with proper mappings
  4. Update dashboard references
  5. Import dashboards

Deployment

  • Includes Dockerfile for Docker containerization
  • Use Python 3.11 and above
  • Dependencies managed through requirementes.txt
@startuml
participant 版本1    as Distro1
actor       版本1运维      as Dev1
control     导出脚本         as Export
collections 数据         as Data
control     导入脚本         as Import
actor       版本2运维      as Dev2
participant 版本2    as Distro2

group Export [导出流程]
    activate Dev1
    Dev1 -> Export: 输入配置，启动导出
    deactivate Dev1
    activate Export
    Export -> Distro1: 通过 OpenAPI 查询实例配置
    deactivate Export
    activate Distro1
    Distro1 -> Export: 查询成功
    deactivate Distro1
    activate Export
    Export -> Data: 保存配置描述文件
    deactivate Export
    activate Data
    Data -> Export: 保存成功
    deactivate Data
    activate Export
    Export -> Dev1: 导出成功
    deactivate Export
    activate Dev1
end
Dev1 -> Dev2: 通过邮件将数据、导入脚本和操作指南发至版本2的运维人员
deactivate Dev1
group Import [导入流程]
    activate Dev2
    Dev2 -> Import: 输入配置，启动导入
    deactivate Dev2
    activate Import
    Import -> Data: 检查并加载数据
    deactivate Import
    activate Data
    Data -> Import: 加载成功
    deactivate Data
    activate Import
    Import -> Distro2: 通过 OpenAPI 配置实例
    deactivate Import
    activate Distro2
    Distro2 -> Import: 配置成功
    deactivate Distro2
    activate Import
    Import -> Dev2: 导入成功
    deactivate Import
    activate Dev2
end
deactivate Dev2
@enduml

 

@startuml
[*] --> Welcome
Welcome --> Export: 1
Welcome --> Import: 2
Welcome --> Welcome: other
Export --> ConfigCheckFailed: rejected
Import --> ConfigCheckFailed: rejected
ConfigCheckFailed -> [*]

Export --> ListDashboards: csv not exists
Export --> ConfirmExport: csv exists
ListDashboards --> ConfirmExport: save to csv
ConfirmExport --> [*]: N
ConfirmExport --> PasrseDashbords : Y
PasrseDashbords --> ExportFinish: success
PasrseDashbords --> ExportFaied: fail
ExportFinish --> [*]
ExportFaied --> [*]

Import --> ConfirmImport: passed
ConfirmImport --> [*]: N 
ConfirmImport --> ImportDatasource: both/datasources
ConfirmImport --> ImportDashboard: dashbords
ImportDatasource --> ImportDashboard: both
ImportDatasource --> ImportFailed: fail
ImportDatasource --> ImportSuccess: dashbords&sucess
ImportDashboard --> ImportSuccess: sucess
ImportDashboard --> ImportFailed: fail
ImportSuccess --> [*]
ImportFailed --> [*]

Welcome: 想要「导入」还是「导出」看板？
Welcome: 1: 导入
Welcome: 2: 导出
Export: 已选择「导出」
Export: 检查导出配置中
Import: 已选择「导入」
Import: 检查导入配置中
ConfigCheckFailed: 配置检查不通过：请参考...
ListDashboards: 已将xx空间的看板列出，请确认
ConfirmExport: 确认要将这些看板导出吗
PasrseDashbords: 正在解析看板描述文件中的数据源
ExportFinish: 导出已完成！请按指南检查文件
ExportFaied: 导出失败！请按指南修复

ConfirmImport: 确认要将这些看板和/或数据源导入吗
ImportDatasource: 正在导入数据源
ImportDashboard: 正在导入看板
ImportFailed: 导入失败！请按指南修复
ImportSuccess: 导入成功！请检查看板是否正常

@enduml