How do I sync a mysql table to a hive table ? (sqoop --incremental lastmodified hive imports is not supported)

51
January 12, 2019, at 05:10 AM

I want to sync a mysql table into hive table. Because records in orders table usually changed in nearly future . I need update them into hive .

For example ,

  1. I dump all mysql data into hive
  2. daily job check the changed record which time_update is in nearly 1 days, and update them into hive table.

I have tried --incremental lastmodified like below

sqoop import \
"-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \
--connect $DB_URL \
--username $USERNAME \
--password $PASSWORD \
--direct \
--fields-terminated-by '\t' \
--target-dir '/data/hive/' \
--delete-target-dir \
--hive-database $HIVE_DB \
--hive-table $HIVE_TABLE \
--hive-import \
--hive-overwrite \
--create-hive-table \
--query 'select * from '$HIVE_TABLE' where $CONDITIONS' \
--split-by id \
-m 6 \
--merge-key id \
--incremental lastmodified \
--check-column time_update \
--last-value "2019-01-01 21:00:00"

Got error --incremental lastmodified option for hive imports is not supported. Please remove the parameter --incremental lastmodified.

What is the proper way to do without --incremental lastmodified option .

READ ALSO
How to execute PHP code with MySQL through command line?

How to execute PHP code with MySQL through command line?

I am trying to execute my PHP script through command line using below command:

22
I want to connect between MYSQL 5.7 with Crystal Rport using Windows Application framework 4

I want to connect between MYSQL 5.7 with Crystal Rport using Windows Application framework 4

Need a help for what are the drivers and version to install and connect MYSQL server 57 with Crystal Report for Windows Application Framework 4

10
SELECT random rows with flexible search query

SELECT random rows with flexible search query

I am trying to create a Flexible Search that retrieves 10,000 random rows out of 3MTried with different syntax like LIMIT, but i cannot make it work, I need to add this query to a Groovy Script

26