Importing data incrementally from RDBMS to hive/hadoop using sqoop -


i have oracle database , need import data hive table. daily import data size around 1 gb. better approach?

if import each day data partition, how can updated values handled?

for example, if imported today's data partition , next day there fields updated new values.

using --lastmodified can values need send updated values new partition or old (already existing) partition?

if send new partition, data duplicated. if want send existing partition, how can achieved?

your option override entire existing partition 'insert overwrite table...'.
question - how far going updating data?
think of 3 approaches u can consider:

  • decide on threshold 'fresh' data. example '14 days backwards' or '1 month backwards'.
    each day running job, override partitions (only ones have updated values) backwards, until threshold decided.
    ~1 gb day should feasible.
    data before decided time not guranteed 100% correct.
    scenario relevant if know fields can changed time window after set.
  • make hive table compatible acid transactions, allowing updates on table.
    split daily job 2 tasks: new data being written run day. updated data need run backwards. sqoop responsible new data. take care of updated data 'manually' (some script generates update statements)
  • don't use partitions based on time. maybe dynamic partitioning more suitable use case.it depends on nature of data being handled.

-

Comments

Post a Comment

Popular posts from this blog

java - SSE Emitter : Manage timeouts and complete() -

i writing web app multiple listeners(evcentsource sse clients js) connected server . want store sse emitter per connected listener : can done in memory or other means assigning id each client able achieve far now question ; how send response/event specific client connected web app ? while doing sseemmiters stored either getting completed or timed out. how prevent ? how keep sseemmiter available open infinite time (till client closes) , send events selectively .
Read more

jquery - uncaught exception: DataTables Editor - remote hosting of code not allowed -

i'm trying implement crud operation using datatableseditor using datatables.but i'm getting error messages: 1)uncaught exception: datatables editor - remote hosting of code not allowed. please see http://editor.datatables.net details on how purchase editor license 2)typeerror: a.editor undefined 3) $.fn.datatable.editor not constructor" what reason? here configuration: <%@ page language="java" contenttype="text/html; charset=iso-8859-1" pageencoding="iso-8859-1"%> <!doctype html public "-//w3c//dtd html 4.01 transitional//en" "http://www.w3.org/tr/html4/loose.dtd"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=iso-8859-1"> <title>insert title here</title> <link rel="stylesheet" type="text/css" href="//cdn.datatables.net/1.10.12/css/jquery.datatables.min.css"> <link re...
Read more

java - How to resolve error - package com.squareup.okhttp3 doesn't exist? -

i trying convert json java object use of gson in maven, following youtube video guidance - https://www.youtube.com/watch?v=vqgghm9pwe0 , theres error occurs when in main class error - package com.squareup.okhttp3 doesn't exist. code below: java package com.codebeasty.json; import com.squareup.okhttp3.okhttpclient; public class main { private static okhttpclient client = new okhttpclient(); public static void main (string [] args) { } } i put in dependency in pom.xml: <dependencies> <dependency> <groupid>com.squareup.okhttp3</groupid> <artifactid>okhttp</artifactid> <version>3.4.2</version> </dependency> <dependency> <groupid>com.google.code.gson</groupid> <artifactid>gson</artifactid> <version>2.8.0</version> </dependency> </dependencies> i dont understand why doesn't recognize com.squareup. there may need download? have downl...
Read more