Every now and then you might get a report that a web service “is slow” and sometimes it is hard to stay focused while trying to identify the root cause. Following the below steps will make your life a little easier when taming web application / REST API performance related complains:
Step 1 Identify the “slowness” – As an old colleague used to say
“We don’t have a log for slow”, so you need to determine the specific action with unacceptable timing. If you get an answer like “Everything is slow!” ask for the most annoying action and focus on it.
Step 2 Determine if it is really slow. Check if there are existing specifications or SLAs on the timing. Sometimes there are actions which are expected to take more time and in this case this should be clearly communicated or acknowledged in UI (progress bars, spinners etc.)
Step 3 Presuming the specific slow operation is identified the real troubleshooting can begin. The key aspect of this approach is to narrow down the possible causes. The total time of the problematic operation can be split and each part examined individually:
Total request time = (server processing time + transmission time) + client processing time
The total time can be determined by observing the network tab in the browser console, API request timing statistics or simply counting the seconds it take for the action to complete. In all cases at least one of the three components will contribute to the significant overall delay and as always the usual suspect is:
Server Processing Time
This is the time required by the back end application to process the request. Most web/app servers log information of per request timing in their access log and observing this information will give you clue if the server is struggling. First you need to check the access log, to see if the request processing time and request URL are being logged for each request and enable them if not. All major web/app servers have this functionality well documented in detail. Below you can find an example configuration snippet for nginx:
log_format custom ‘$remote_addr – $remote_user “$request” ‘
‘$status $body_bytes_sent “$http_referer” ‘
access_log /var/log/nginx/access.log custom;
The above lines will tell nginx to log the processing time of each served request. Example:
220.127.116.11 – – “POST /icm/collaboration-connector/newRoom HTTP/1.1″ 502 166 “-” “Squared Scheduler/1.0” Rtime=”3.001″
The log entry shows that a post request from IP 18.104.22.168 for the location /icm/collaboration-connector/newRoom took 3 seconds to complete.
You can now search the logs by client IP address and request URL to locate the according entries and check how long it took the server to process them.
NOTE: The request processing time in access log also includes the request and response header and body transmission times. So larger request/response size can cause a delay over a slower network. This case can easily be diagnosed by running some speed tests from the client side. In addition the case of slow network connectivity would lead to a performance penalty to all web sites / applications.
Client processing time
Verdict: While the above is a high level guidance, the essence of this approach is to help you stay on track and prevent the chasing of rabbits down rabbit holes. Remember, pinpoint specific actions or operations and troubleshoot only them – usually such approach reveals the main bottlenecks of a system.