We identified the root cause of the issue. At 3:25pm EST a developer was researching the correlation between accounts and database sizes in order to improve our backups system. The question asked of the database was not properly optimized and, coupled with a bug in database management software, caused too much load on the database.
As a result, Follow Up Boss web app, mobile apps and API were unavailable from 3:27pm to 3:47pm EST. Leads coming via Inbox Lead Processing (connected Google address), @followupboss.me addresses and Zillow Tech Connect were delayed for the time of the outage and should have been processed by now. Leads coming via API integrations such as IDX websites, Curaytor, Ylopo, Sierra Interactive may not be processed. We recommend you login into those systems directly and check for leads arrived between 3:27pm and 3:47pm EST to make sure no leads are missed.
We sincerely apologize for the service interruption. We are taking multiple steps to ensure this problem doesn't happen again. These include upgrading database management software to a new version and preventing running similar queries on production databases.
Posted Nov 09, 2018 - 16:44 EST
Monitoring
An issue was identified in our database tier which prompted timeouts in our web application. We have resolved the immediate issue but are still searching for the root cause. We apologize for the inconvenience and appreciate your patience as we investigate the situation further.
Posted Nov 09, 2018 - 15:54 EST
Investigating
At approximately 3:30 PM Eastern time, we were alerted to issues accessing our website. Our technicians are investigating the issue and will publish updates when we have more information about the issue.
Posted Nov 09, 2018 - 15:33 EST
This incident affected: Follow Up Boss Web Application.