java - Spark SQL - DataFrameReader load method with where condition -

-

i trying use dataframereader.load("table name") load hive table records , return dataframe.

but dont want load entire records, wanted fetch records specific date (which 1 of field in hive table).

if add condition in returned dataframe, load entire table first filter records based on date?

because hive tables huge , partitioned based on date field.

basically want achieve select * table date='date' using load method without loading entire table.

recent versions of spark support feature called "predicate push-down". want: pushes, possible, sql clauses source. i'm not sure if predicate push-down works hive data source (it works parquet, jdbc , others sources). see does spark predicate pushdown work jdbc?


Comments

Post a Comment

Popular posts from this blog

php - Passing multiple values in a url using checkbox -

-
Image
i know may answered not able find related this. here thing struggling :- i have 2 value passing in check box :- <input name='class[]' type='checkbox' class='checkbox' value='".$x_value['type1'].":".$x_value['type2']."'> on running :- <input name="class[]" type="checkbox" class="checkbox" value="2681:14175"> so want how can pass these value separated ":" in url. something :- http://domainname.com/approve.php?type1=2681&type2=14175 purpose :- actually getting value api response curl , generating multiple checkboxes. each checkbox contains 2 value separated ":". now want run separate api in different file values url. update : below complete code :- <?php foreach($apiresponse['response']['data'] $x =>$x_value) { foreach($x_value $x => $x_value) { echo "...
Read more

compilation - PHP install fails on Ubuntu 14 (make: *** [sapi/cli/php] Error 1) PHP 5.6.20 -

-
i using following configuration: ./configure --with-libdir=lib --with-config-file-path=/etc/ --with-config-file-scan-dir=/etc/php5/mods-available --with-pear --enable-cli --enable-intl --with-icu-dir=/opt/icu4c --enable-bcmath --disable-cgi --enable-fpm --with-zlib --with-openssl --with-kerberos --with-bz2 --with-curl --enable-ftp --enable-zip --enable-exif --with-gd --with-jpeg-dir=/usr/lib64 --with-freetype-dir=/usr/lib64 --enable-gd-native-ttf --with-gettext --with-gmp --with-mhash --with-iconv --with-imap --with-mysql --with-imap-ssl --enable-sockets --enable-soap --with-xmlrpc --with-mcrypt --enable-mbstring --enable-embedded-mysqli --with-mysqli=mysqlnd --with-mysql-sock --with-sqlite3 --with-pdo-mysql --with-pdo-mysql=mysqlnd --with-pdo-sqlite --enable-phar --enable-pcntl and make fails after sometime giving following error: collect2: error: ld returned 1 exit status make: *** [sapi/cli/php] error 1 i searched , docs tell using make clean, did not , ‘-lresolve’ in ma...
Read more

sql - Postgresql tables exists, but getting "relation does not exist" when querying -

-
Image
i have postgresql db number of tables. if query: select column_name information_schema.columns table_name="my_table"; i list of columns returned properly. however, when query: select * "my_table"; i error: (programmingerror) relation "my_table" not exist 'select *\n "my_table"\n' {} any thoughts on why can columns, can't query table? goal able query table. you have include schema if isnt public one select * <schema>."my_table" or can change default schema show search_path; set search_path my_schema; check table schema here select * information_schema.columns for example if table on default schema public both works ok select * parroquias_region select * public.parroquias_region but sectors need specify schema select * map_update.sectores_point
Read more