i designed data warehousing application, struggle poor performance when fetching data source , saving them db. - approximately 150 kb/s. because of limitations imposed customer, forced use django on 64bit win machine , save data ms sql express (exact versions below). using django-mssql (1.7) backend. original data stored in .dbf file (visual foxpro), dbfread returns each row file python dict (this not issue, tested running reader discarding data). dictionary checked data quality (function sanitize_value_for_db() below), data copied django data model (attributes of object populated; tables wide, hence each table/object has 100 columns/attributes) , objects saved db. using django objects.bulk_create() (in batches of 50-100). i run code through profiler using cprofile , pstats modules. results below. see of time spent in pywin. have no clue if there can do. hints or opinions appreciated. thanks. configuration: xeon e5-2403 v2 @ 1,8 ghz, 30 gb ram windows server 2012...
Comments
Post a Comment