hadoop - Sqoop incremental export using hcatalog? -
is there way use sqoop incremental exports ? using hcatalog integration sqoop.i tried using --last-value, --check-column options used incremental import, sqoop gave me error options invalid.
i have not seen incremental sqoop export arguments. other way try create contol_table in hive keep log of table name & timestamp when last exported every time.
create table if not exists control_table ( table_name string, export_date timestamp ); insert control_table 'export_table1' table_name, from_unixtime(unix_timestamp()) export_date control_table;
if export_table1 table want export incrementally , assuming if have executed above 2 statements.
--execute below @ once --get timestamp when table last executed create temporary table control_table_now select table_name, max(export_date) last_export_date control_table group table_name; --get incremental rows create table new_export_table1 select field1, field2, field3, .... timestamp1 export_table1 e, control_table_now c c.table_name = 'export_table1' , e.timestamp1 >= c.last_export_date; --append control_table next process insert control_table 'export_table1' table_name, from_unixtime(unix_timestamp()) export_date control_table;
now, export new_export_table1 table incrementally created using sqoop export command.
Comments
Post a Comment