performance - How to speed up Pandas multilevel dataframe shift by group? -

-

i trying shift pandas dataframe column data group of first index. here demo code:

 in [8]: df = mul_df(5,4,3)  in [9]: df out[9]:                  col000  col001  col002 stk_id rpt_date a0000  b000     -0.5505  0.7445 -0.3645        b001      0.9129 -1.0473 -0.5478        b002      0.8016  0.0292  0.9002        b003      2.0744 -0.2942 -0.7117 a0001  b000      0.7064  0.9636  0.2805        b001      0.4763  0.2741 -1.2437        b002      1.1563  0.0525 -0.7603        b003     -0.4334  0.2510 -0.0105 a0002  b000     -0.6443  0.1723  0.2657        b001      1.0719  0.0538 -0.0641        b002      0.6787 -0.3386  0.6757        b003     -0.3940 -1.2927  0.3892 a0003  b000     -0.5862 -0.6320  0.6196        b001     -0.1129 -0.9774  0.7112        b002      0.6303 -1.2849 -0.4777        b003      0.5046 -0.4717 -0.2133 a0004  b000      1.6420 -0.9441  1.7167        b001      0.1487  0.1239  0.6848        b002      0.6139 -1.9085 -1.9508        b003      0.3408 -1.3891  0.6739  in [10]: grp = df.groupby(level=df.index.names[0])  in [11]: grp.shift(1) out[11]:                  col000  col001  col002 stk_id rpt_date a0000  b000         nan     nan     nan        b001     -0.5505  0.7445 -0.3645        b002      0.9129 -1.0473 -0.5478        b003      0.8016  0.0292  0.9002 a0001  b000         nan     nan     nan        b001      0.7064  0.9636  0.2805        b002      0.4763  0.2741 -1.2437        b003      1.1563  0.0525 -0.7603 a0002  b000         nan     nan     nan        b001     -0.6443  0.1723  0.2657        b002      1.0719  0.0538 -0.0641        b003      0.6787 -0.3386  0.6757 a0003  b000         nan     nan     nan        b001     -0.5862 -0.6320  0.6196        b002     -0.1129 -0.9774  0.7112        b003      0.6303 -1.2849 -0.4777 a0004  b000         nan     nan     nan        b001      1.6420 -0.9441  1.7167        b002      0.1487  0.1239  0.6848        b003      0.6139 -1.9085 -1.9508

the mul_df() code attached here : how speed pandas multilevel dataframe sum?

now want grp.shift(1) big dataframe.

in [1]: df = mul_df(5000,30,400) in [2]: grp = df.groupby(level=df.index.names[0]) in [3]: timeit grp.shift(1) 1 loops, best of 3: 5.23 s per loop

5.23s slow. how speed ?

(my computer configuration is: pentium dual-core t4200@2.00ghz, 3.00gb ram, windowxp, python 2.7.4, numpy 1.7.1, pandas 0.11.0, numexpr 2.0.1 , anaconda 1.5.0 (32-bit))

how shift total dataframe object , set first row of every group nan?

dfs = df.shift(1) dfs.iloc[df.groupby(level=0).size().cumsum()[:-1]] = np.nan

Comments

Post a Comment

Popular posts from this blog

java - nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet Hibernate+SpringMVC -

-
i trying fetch records database , having above error this classs appropriate getters , setters public class user { private int id; private string username; private string password; private string email; i have create database , inserted data below configuration <bean id="datasource" class="org.apache.commons.dbcp2.basicdatasource" destroy-method="close"> <property name="driverclassname" value="com.mysql.jdbc.driver"/> <property name="url" value="jdbc:mysql://localhost:3306/usersdb"/> <property name="username" value="root"/> <property name="password" value=""/> </bean> this complete stacktrace severe: servlet.service() servlet [appservlet] in context path [/spring] threw exception [request processing failed; nested exception org.hibernate.exception.sqlgrammarexception: not ...
Read more

sql - Postgresql tables exists, but getting "relation does not exist" when querying -

-
Image
i have postgresql db number of tables. if query: select column_name information_schema.columns table_name="my_table"; i list of columns returned properly. however, when query: select * "my_table"; i error: (programmingerror) relation "my_table" not exist 'select *\n "my_table"\n' {} any thoughts on why can columns, can't query table? goal able query table. you have include schema if isnt public one select * <schema>."my_table" or can change default schema show search_path; set search_path my_schema; check table schema here select * information_schema.columns for example if table on default schema public both works ok select * parroquias_region select * public.parroquias_region but sectors need specify schema select * map_update.sectores_point
Read more

asp.net mvc - breakpoint on javascript in CSHTML? -

-
i have .cshtml file has javascript tag. within script tag have few lines of javascript. first line of javascript refers property in @viewbag. other javascript lines plain script without mvc related references. observation: can place breakpoints (visual studio 2015) on each of these javascript lines. however, breakpoints has white dot in center of breakpoint symbol (red circle) except line @viewbag reference (this particular line not have white dot in center). it seems breakpoint without white dot hit during run-time. question: can explain going on here? white dot mean? why plain red breakpoint line line hitting breakpoint? the visual studio debugger expecting debug actual server-side code within .cshtml file opposed client-side javascript within it. iirc, visual studio allow debug javascript code expected in internet explorer (and possibly edge), other browsers want use either third-party tool or developer tools (f12) within browser. an easy approach take advant...
Read more