How to Connect Python to Spark Session and Keep RDDs Alive -


how small python script hook existing instance of spark , operations on existing rdds?

i'm in stages of working spark on windows 10, trying scripts on "local" instance. i'm working latest stable build of spark (spark 2.0.1 hadoop 2.7). i've installed , set environment variables hadoop 2.7.3. i'm experimenting both pyspark shell , visual studio 2015 community python.

i'm trying build large engine, on i'll run individual scripts load, massage, format, , access data. i'm sure there's normal way that; isn't point of spark?

anyway, here's experience have far. expected. when build small spark script in python , run using visual studio, script runs, job, , exits. in process of exiting, exits spark context using.

so had following thought: if started persistent spark context in pyspark , set sparkconf , sparkcontext in each python script connect spark context? so, looking online defaults pyspark, tried following:

conf = sparkconf().setmaster("local[*]").setappname("pysparkshell") sc = sparkcontext(conf = conf)

i started pyspark. in separate script in visual studio, used code sparkcontext. loaded text file rdd named rddfromfilename . couldn't access rdd in pyspark shell once script had run.

how start persistent spark context, create rdd in in 1 python script, , access rdd subsequent python scripts? particularly in windows?

there no solution in spark. may consider:

i think out of these zeppelin officially supports windows.


-

Comments

Post a Comment

Popular posts from this blog

java - SSE Emitter : Manage timeouts and complete() -

i writing web app multiple listeners(evcentsource sse clients js) connected server . want store sse emitter per connected listener : can done in memory or other means assigning id each client able achieve far now question ; how send response/event specific client connected web app ? while doing sseemmiters stored either getting completed or timed out. how prevent ? how keep sseemmiter available open infinite time (till client closes) , send events selectively .
Read more

jquery - uncaught exception: DataTables Editor - remote hosting of code not allowed -

i'm trying implement crud operation using datatableseditor using datatables.but i'm getting error messages: 1)uncaught exception: datatables editor - remote hosting of code not allowed. please see http://editor.datatables.net details on how purchase editor license 2)typeerror: a.editor undefined 3) $.fn.datatable.editor not constructor" what reason? here configuration: <%@ page language="java" contenttype="text/html; charset=iso-8859-1" pageencoding="iso-8859-1"%> <!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http://www.w3.org/tr/html4/loose.dtd"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> <title>insert title here</title> <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.12/css/jquery.datatables.min.css"> <link re...
Read more

java - How to resolve error - package com.squareup.okhttp3 doesn't exist? -

i trying convert json java object use of gson in maven, following youtube video guidance - https://www.youtube.com/watch?v=vqgghm9pwe0 , theres error occurs when in main class error - package com.squareup.okhttp3 doesn't exist. code below: java package com.codebeasty.json; import com.squareup.okhttp3.okhttpclient; public class main { private static okhttpclient client = new okhttpclient(); public static void main (string [] args) { } } i put in dependency in pom.xml: <dependencies> <dependency> <groupid>com.squareup.okhttp3</groupid> <artifactid>okhttp</artifactid> <version>3.4.2</version> </dependency> <dependency> <groupid>com.google.code.gson</groupid> <artifactid>gson</artifactid> <version>2.8.0</version> </dependency> </dependencies> i dont understand why doesn't recognize com.squareup. there may need download? have downl...
Read more