https://melotel.com/wp-content/uploads/2020/05/inner-banner2.png
The Most Reliable & Trusted
VOIP Service Provider
in Canada
August 3, 2012

Web Server Incident Report – August 2, 2012

Category:

The short description: The server backups took longer than expected and as the web traffic started to grow the server got overloaded and crashed.

TIMELINE for Thursday, August 2nd, 2012, 10:15-19:20

  • 10:15 The first internal monitoring alert arrived
  • 10:17 First customer e-mail arrived
  • 10:18 First attempt to log in to the server
  • 10:18 Acknowledged the customer e-mail
  • 10:25 The first SSH and Remote console login attempts are unsuccessful
  • 10:38 Since the remote connection was not possible, the server was rebooted
  • 10:40 The server did not start-up after the reboot due to file system consistency errors
  • 10:45 MeloTel technicians launch a file consistency check and repair tool
  • 13:30 The file consistency repair was finished
  • 13:35 Database issues noticed on some accounts
  • 13:40 Started to fix database files
  • 13:47 Identified the corrupt DB`s
  • 14:10 Continued to fix DB files
  • 15:45 Noticed that tables that use inno-db engine were corrupted
  • 16:50 Made backups of the inno-db corrupted files
  • 17:10 Started inno-db files recovery
  • 18:57 Finished the recovery and started testing the databases and all websites and CRMs
  • 19:20 Finished testing and notified our customers

ROOT CAUSE ANALYSIS

The backups that ran simultaneously with the traffic crashed the server. The backup process was too long and it was extended to the business hours.

CORRECTIVE/PREVENTATIVE ACTIONS

MeloTel has made major improvements to the backup process.

0 Comment

Here are a few testimonials from what our customers have to say:

Danielle Cameron
Heidi Kaufmann
Charles Murray
Rashalee Green
Fouad Shuhaiber
Dr. Martin Jugenburg
Wayne Goldstein
Danielle Cameron (VP Marketing)Imprezzio Marketing

“Imprezzio Marketing has been with MeloTel for 4 years and their service has been outstanding over the years. Their customer support team is always extremely helpful and always responds and resolves any issues that we have. Thank you MeloTel for providing us with such great service. READ MORE"

Heidi Kaufmann (Manager )TK Enterprises

“When TK Enterprises found MeloTel 2 years ago, it was in a crisis situation. Our previous VoIP provider had left us in the lurch and within days, MeloTel had us back up and running and we’ve never looked back since. We started with VoIP services and have worked with them to bring our company’s technology levels up to be more cutting edge and competitive while reducing costs and mitigating risk the entire time. READ MORE"

Charles Murray (CEO)Benjamin Verde

“Benjamin Verde Incorporated has been a loyal MeloTel customer since 2013. We are able to provide a variety of Artist and Entertainment related solutions such as Web Hosting, Artist Marketing and Artist Funding to clients across Canada. By utilizing MeloTel’s robust products and services our agents are able to provide fast and efficient customer care to various radio stations, record companies, independent artist as well as popular international brands via Benjamin Verde’s VXCO.net portal. READ MORE"

Rashalee Green (VP of Operations)Synergy Marketing

“As the leaders in promotional advertising with over 25 years experience we strive to satisfy all of our clients’ needs when it comes to marketing them and their businesses. We go over and above to make sure that all of their advertising demands are met with no exceptions. MeloTel and Synergy have been in business together since providing exceptional VoIP and technical services. READ MORE"

Fouad Shuhaiber (CEO)National Projects

“I have been a MeloTel customer since 2009 and wanted to tell you how much my whole office appreciates your services. Life becomes a lot easier when someone is there to help when we need it. That kind of service is why we will remain a loyal customer for many years to come. Thank you! READ MORE"

Dr. Martin Jugenburg (Surgeon)Toronto Cosmetic Surgery Institute

“Toronto Cosmetic Surgery Institute is a plastic surgery clinic, located in downtown Toronto. On site we have a fully functional, state of the art surgical facility. MeloTel assisted us when we went paperless a couple years ago in replacing and restructuring our computer hardware and network. They have been providing us with IT support ever since. Whether it be remote sessions that are required, or on site visits, the service MeloTel provides is always quick and efficient. READ MORE"

Wayne Goldstein (CEO)Global Mentoring Solutions

“Global Mentoring Solutions (GMS) is a North American based, white labelled outsourced help desk provider that partners with major ISPs, eLearning Support Providers and mid-market IT Service Providers. With MeloTel’s telephone infrastructure services GMS is able to provide support for over 8,000 technical support interactions per month! GMS has been relying on MeloTel’s expertise and professional support services since 2010. READ MORE"

Request Information

Connect With Us

Back