Nginx / Apache / WSGI / Django - Unexpected raising 500 errors (499 on Nginx) -

-

i have problems django server on webfaction hosting. request (about 1.000 request 500.000 per day) can see errors in access logs. in nginx log can see requests code "499", in apache log have "500" error code.

weird not see problems in django logs (usually receiving mail excepting stacktrace). can me, can cause strange behaviour?

i start investigate logs after receiving problems exceeding hosting memory limit. seems app has memory leaks, because can see used memory rising. tryed quickfix optimize wsgi processes, thread , try set maximum-request param. after investigate cause of memory leak.

my infrastructure following:

  • nginx - frontend server (this operated webfaction)
  • apache/2.4.16, mod_wsgi 4.4.13/python 3.4
  • django 1.8.12

my httpd.conf following:

serverroot "/home/quick/webapps/citaty/apache2"  loadmodule authz_core_module modules/mod_authz_core.so loadmodule dir_module        modules/mod_dir.so loadmodule env_module        modules/mod_env.so loadmodule log_config_module modules/mod_log_config.so loadmodule mime_module       modules/mod_mime.so loadmodule rewrite_module    modules/mod_rewrite.so loadmodule setenvif_module   modules/mod_setenvif.so loadmodule wsgi_module       modules/mod_wsgi.so loadmodule unixd_module      modules/mod_unixd.so  logformat "%{x-forwarded-for}i %l %u %t %{host}i \"%r\" %>s %b \"%{referer}i\" \"%{user-agent}i\" %t %d" combined customlog /home/quick/logs/user/access_citaty.log combined errorlog /home/quick/logs/user/error_citaty.log  listen 21628 keepalive off setenvif x-forwarded-ssl on https=1 serverlimit 1 startservers 1 maxrequestworkers 10 minsparethreads 1 maxsparethreads 5 threadsperchild 10 maxrequestsperchild 1000  wsgirestrictembedded on wsgilazyinitialization on  <virtualhost *>     servername domain.com      customlog /home/quick/logs/user/access_citaty.log combined     errorlog /home/quick/logs/user/error_citaty.log      wsgidaemonprocess citaty processes=5 threads=1 maximum-requests=1000 python-path=/home/quick/webapps/citaty:/home/quick/webapps/citaty/lib/python3.4/site-packages display-name=%{group}     wsgiprocessgroup citaty     wsgiscriptalias / /home/quick/webapps/citaty/apache2/conf/citaty.wsgi  </virtualhost>  <virtualhost *>     servername domain2.com      customlog /home/quick/logs/user/access_citaty_laska.log combined     errorlog /home/quick/logs/user/error_citaty_laska.log      wsgidaemonprocess citaty_laska processes=1 threads=1 maximum-requests=1000 python-path=/home/quick/webapps/citaty:/home/quick/webapps/citaty/lib/python3.4/site-packages display-name=%{group}     wsgiprocessgroup citaty_laska     wsgiscriptalias / /home/quick/webapps/citaty/apache2/conf/citaty_laska.wsgi </virtualhost>

my django logging settings following:

logging = {     'version': 1,     'disable_existing_loggers': false,     'filters': {         'require_debug_false': {             '()': 'django.utils.log.requiredebugfalse',         },         'require_debug_true': {             '()': 'django.utils.log.requiredebugtrue',         },     },     'handlers': {         'file': {             'level': 'info',             'filters': ['require_debug_false'],             'class': 'logging.filehandler',             'filename': '/home/quick/logs/user/django_citaty.log',         },         'file_jobs': {             'class': 'logging.filehandler',             'filename': '/home/quick/logs/user/django_jobs.log',         },         'mail_admins': {             'level': 'error',             'filters': ['require_debug_false'],             'class': 'django.utils.log.adminemailhandler'         },         'console': {             'class': 'logging.streamhandler',         },     },     'loggers': {         'django': {             'handlers': ['file', 'mail_admins'],         },         'py.warnings': {             'handlers': ['file'],         },         'citaty.sources.crawlers.base': {             'level': 'info',             'handlers': ['file_jobs', 'console'],         },      }, }

i used face problem month back. of sudden nginx start throwing 499 error code.

when client makes request, nginx asks uwsgi response, uwsgi takes long respond, user thinks request slow , tries refresh or makes call, nginx throws 499 error code.

possible reasons:-

  1. uwsgi workers consume quite bit of ram. check how many workers there in uwsgi configuration file. make sure limit uwsgi worker process appropriate number based on system settings.
  2. increase nginx worker connections. default low. causes connections drops result in 500 errors too. check connection_timeout settings in nginx. 60 secs. can reduce 25 sec or make sure nginx ready accept more connections.

i fixed issue increasing server ram , added load balancer make sure there no heavy load on 1 server. also, made changes uwsgi ini file , nginx.conf

below uwsgi settings file-

[uwsgi] project =xxxxxx base = /home/ubuntu  chdir = %(base)/xxxxxxx/server home = %(base)/envs/api module = %(project).wsgi:application  master = true cheaper-algo = spare cheaper = 2 cheaper-initial = 3 cheaper-step = 2 processes = 20 workers = 20 max-requests = 1000  socket = %(base)/xxxx/server/%(project).sock chmod-socket = 664 vacuum = true logto=/tmp/uwsgi.log

my nginx.conf-

user www-data; worker_processes 4; pid /run/nginx.pid;  events {     worker_connections 2048;     # multi_accept on; }

bottomline - memory issue. recheck code memory leaks. make sure debug=false in settings.py in production. use cache reduce server response times.


Comments

Post a Comment

Popular posts from this blog

java - nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet Hibernate+SpringMVC -

-
i trying fetch records database , having above error this classs appropriate getters , setters public class user { private int id; private string username; private string password; private string email; i have create database , inserted data below configuration <bean id="datasource" class="org.apache.commons.dbcp2.basicdatasource" destroy-method="close"> <property name="driverclassname" value="com.mysql.jdbc.driver"/> <property name="url" value="jdbc:mysql://localhost:3306/usersdb"/> <property name="username" value="root"/> <property name="password" value=""/> </bean> this complete stacktrace severe: servlet.service() servlet [appservlet] in context path [/spring] threw exception [request processing failed; nested exception org.hibernate.exception.sqlgrammarexception: not ...
Read more

sql - Postgresql tables exists, but getting "relation does not exist" when querying -

-
Image
i have postgresql db number of tables. if query: select column_name information_schema.columns table_name="my_table"; i list of columns returned properly. however, when query: select * "my_table"; i error: (programmingerror) relation "my_table" not exist 'select *\n "my_table"\n' {} any thoughts on why can columns, can't query table? goal able query table. you have include schema if isnt public one select * <schema>."my_table" or can change default schema show search_path; set search_path my_schema; check table schema here select * information_schema.columns for example if table on default schema public both works ok select * parroquias_region select * public.parroquias_region but sectors need specify schema select * map_update.sectores_point
Read more

asp.net mvc - breakpoint on javascript in CSHTML? -

-
i have .cshtml file has javascript tag. within script tag have few lines of javascript. first line of javascript refers property in @viewbag. other javascript lines plain script without mvc related references. observation: can place breakpoints (visual studio 2015) on each of these javascript lines. however, breakpoints has white dot in center of breakpoint symbol (red circle) except line @viewbag reference (this particular line not have white dot in center). it seems breakpoint without white dot hit during run-time. question: can explain going on here? white dot mean? why plain red breakpoint line line hitting breakpoint? the visual studio debugger expecting debug actual server-side code within .cshtml file opposed client-side javascript within it. iirc, visual studio allow debug javascript code expected in internet explorer (and possibly edge), other browsers want use either third-party tool or developer tools (f12) within browser. an easy approach take advant...
Read more