join - IndexedRDD used in streaming context? -


i want use fast join operation in spark streaming context, such join b, fixed dataset reading file, b small streaming rdd read socket. i've tried common way provided spark, 5,000,000 rdd joining 10 streaming rdd costs 4 seconds. later i've tried using indexedrdd, can't make it. have following questions:

  1. is 4 seconds slow? can use performance tuning method such broadcast join improve? if slow, why? heard rdd's join operation linear search, true?

  2. can indexedrdd's join operation faster common way?

  3. how use indexedrdd in streaming context? i've tried way:

    streaming_rdd.transform{ rdd =>                          indexed_data.innerjoin(indexedrdd(rdd)){(id, a, b) => (a, b)}

    it pass compile when running got error: 

    java.lang.classcastexception: scala.collection.immutable.$colon$colon cannot cast [lscala.tuple2;

    i don't know if proper way use indexedrdd, , don't know caused error either. can 1 me?


-

Comments

Post a Comment

Popular posts from this blog

jquery - uncaught exception: DataTables Editor - remote hosting of code not allowed -

i'm trying implement crud operation using datatableseditor using datatables.but i'm getting error messages: 1)uncaught exception: datatables editor - remote hosting of code not allowed. please see http://editor.datatables.net details on how purchase editor license 2)typeerror: a.editor undefined 3) $.fn.datatable.editor not constructor" what reason? here configuration: <%@ page language="java" contenttype="text/html; charset=iso-8859-1" pageencoding="iso-8859-1"%> <!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http://www.w3.org/tr/html4/loose.dtd"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> <title>insert title here</title> <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.12/css/jquery.datatables.min.css"> <link re...
Read more

python - multiprocessing pool with map -

i try multiprocess code this: param_seq = [] pool = pool(multiprocessing.cpu_count()-1) def work(param_set): (_, v) = experiment_sim(param_set[0], param_set[1]) v_dev = np.subtract(exp_v, v) return - np.sum(np.square(v_dev)) / (2 * noise_sigma ** 2) x in ra.values: y in gpas.values: param_seq.append((x, y)) likelihood = pool.map(work, param_seq) and looks working fine, work, in , stops , nothing , have stop program. following message occurs: process poolworker-8: process poolworker-10: process poolworker-13: process poolworker-14: process poolworker-11: process poolworker-12: traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): traceback (most recent call last): file "/usr/local/cellar/python/2.7.11/frameworks/python.framework/versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap file ...
Read more

python - Poor performance of db. inserts using django-mssql -

i designed data warehousing application, struggle poor performance when fetching data source , saving them db. - approximately 150 kb/s. because of limitations imposed customer, forced use django on 64bit win machine , save data ms sql express (exact versions below). using django-mssql (1.7) backend. original data stored in .dbf file (visual foxpro), dbfread returns each row file python dict (this not issue, tested running reader discarding data). dictionary checked data quality (function sanitize_value_for_db() below), data copied django data model (attributes of object populated; tables wide, hence each table/object has 100 columns/attributes) , objects saved db. using django objects.bulk_create() (in batches of 50-100). i run code through profiler using cprofile , pstats modules. results below. see of time spent in pywin. have no clue if there can do. hints or opinions appreciated. thanks. configuration: xeon e5-2403 v2 @ 1,8 ghz, 30 gb ram windows server 2012...
Read more