postgresql - python peewee multiprocessing pool error -

-

stack: python3.4, postgresql 9.4.7, peewee 2.8.0, psycopg2 2.6.1 (dt dec pq3 ext lo64)

i have need able talk(select, inserts, update) postgresql database in each worker. using pythons multiprocessing pool create 10 workers , each 1 makes curl call talks database based on finds.

after reading few threads on internets thought connection pool way go. placed code below atop models.py file. have doubts connections pools because understanding reusing database connections across threads no no.

db = pooledpostgresqlextdatabase(     'uc',     max_connections=32,     stale_timeout=300,  # 5 minutes.     **{'password': cfg['psql']['pass'],         'port': cfg['psql']['port'],         'register_hstore':false,        'host': cfg['psql']['host'],         'user': cfg['psql']['user']})

on question now. getting random sql errors when talking database workers. before introduced peewee mix using "psycopg2" library without wrapper. creating new database connection per worker. there no errors.

a sample error is:

multiprocessing.pool.remotetraceback:  """ traceback (most recent call last):   file "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql     self.commit()   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3468, in commit     self.get_conn().commit() psycopg2.databaseerror: error no message libpq  during handling of above exception, exception occurred:  traceback (most recent call last):   file "/usr/lib/python3.4/multiprocessing/pool.py", line 119, in worker     result = (true, func(*args, **kwds))   file "/usr/lib/python3.4/multiprocessing/pool.py", line 44, in mapstar     return list(map(*args))   file "/home/dan/dev/link-checker/crawler/manager.py", line 17, in startworker     wrk.perform()   file "/home/dan/dev/link-checker/crawler/worker.py", line 49, in perform     self.pullurls()   file "/home/dan/dev/link-checker/crawler/worker.py", line 63, in pullurls     newurldict = urlmanager.createurlwithinprogress(self._url['crawl'], source_url, self._url['base'])   file "/home/dan/dev/link-checker/crawler/models.py", line 152, in createurlwithinprogress     newurl = url.create(**newurldict)   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 4494, in create     inst.save(force_insert=true)   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 4680, in save     pk_from_cursor = self.insert(**field_dict).execute()   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3213, in execute     cursor = self._execute()   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 2628, in _execute     return self.database.execute_sql(sql, params, self.require_commit)   file "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql     self.commit()   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3285, in __exit__     reraise(new_type, new_type(*exc_args), traceback)   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 127, in reraise     raise value.with_traceback(tb)   file "/usr/local/lib/python3.4/dist-packages/playhouse/postgres_ext.py", line 377, in execute_sql     self.commit()   file "/usr/local/lib/python3.4/dist-packages/peewee.py", line 3468, in commit     self.get_conn().commit() peewee.databaseerror: error no message libpq

i tailed postgresql file , saw:

2016-04-19 20:34:23 edt [26824-3] uc_user@uc warning:  there transaction in progress 2016-04-19 20:34:23 edt [26824-4] uc_user@uc warning:  there transaction in progress 2016-04-19 20:34:23 edt [26824-5] uc_user@uc warning:  there no transaction in progress 2016-04-19 20:34:23 edt [26824-6] uc_user@uc warning:  there transaction in progress 2016-04-19 20:34:23 edt [26824-7] uc_user@uc warning:  there no transaction in progress 2016-04-19 20:34:23 edt [26824-8] uc_user@uc warning:  there transaction in progress 2016-04-19 20:34:23 edt [26824-9] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-1] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-2] uc_user@uc warning:  there no transaction in progress 2016-04-19 20:35:14 edt [26976-3] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-4] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-5] uc_user@uc warning:  there no transaction in progress 2016-04-19 20:35:14 edt [26976-6] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-7] uc_user@uc warning:  there no transaction in progress 2016-04-19 20:35:14 edt [26976-8] uc_user@uc warning:  there transaction in progress 2016-04-19 20:35:14 edt [26976-9] uc_user@uc warning:  there no transaction in progress

my hunch connection pool , multiprocessing don't go together. has done without errors , if so, can point me example or give me piece of advice works?

do need explicitly create new connection peewee inside worker or there easier way use peewee multiprocessing pool library.

thanks answers , reading.

i got working, code in models.py file going used workers. wrapped in "with db.execution_context ctx" described on page:

http://docs.peewee-orm.com/en/latest/peewee/database.html#advanced-connection-management


Comments

Post a Comment

Popular posts from this blog

java - nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet Hibernate+SpringMVC -

-
i trying fetch records database , having above error this classs appropriate getters , setters public class user { private int id; private string username; private string password; private string email; i have create database , inserted data below configuration <bean id="datasource" class="org.apache.commons.dbcp2.basicdatasource" destroy-method="close"> <property name="driverclassname" value="com.mysql.jdbc.driver"/> <property name="url" value="jdbc:mysql://localhost:3306/usersdb"/> <property name="username" value="root"/> <property name="password" value=""/> </bean> this complete stacktrace severe: servlet.service() servlet [appservlet] in context path [/spring] threw exception [request processing failed; nested exception org.hibernate.exception.sqlgrammarexception: not ...
Read more

sql - Postgresql tables exists, but getting "relation does not exist" when querying -

-
Image
i have postgresql db number of tables. if query: select column_name information_schema.columns table_name="my_table"; i list of columns returned properly. however, when query: select * "my_table"; i error: (programmingerror) relation "my_table" not exist 'select *\n "my_table"\n' {} any thoughts on why can columns, can't query table? goal able query table. you have include schema if isnt public one select * <schema>."my_table" or can change default schema show search_path; set search_path my_schema; check table schema here select * information_schema.columns for example if table on default schema public both works ok select * parroquias_region select * public.parroquias_region but sectors need specify schema select * map_update.sectores_point
Read more

asp.net mvc - breakpoint on javascript in CSHTML? -

-
i have .cshtml file has javascript tag. within script tag have few lines of javascript. first line of javascript refers property in @viewbag. other javascript lines plain script without mvc related references. observation: can place breakpoints (visual studio 2015) on each of these javascript lines. however, breakpoints has white dot in center of breakpoint symbol (red circle) except line @viewbag reference (this particular line not have white dot in center). it seems breakpoint without white dot hit during run-time. question: can explain going on here? white dot mean? why plain red breakpoint line line hitting breakpoint? the visual studio debugger expecting debug actual server-side code within .cshtml file opposed client-side javascript within it. iirc, visual studio allow debug javascript code expected in internet explorer (and possibly edge), other browsers want use either third-party tool or developer tools (f12) within browser. an easy approach take advant...
Read more