Canadian TV, Computing and Home Theatre Forums banner
21 - 40 of 79 Posts

·
Registered
Joined
·
1,442 Posts
Someone in the IT dept needs to get the boot. The second time in two years a software update knocked out Rogers. Obviously no proper testing and no proper back out plan.

As for the ATMs etc that went down that is partly their fault. They should have a backup plan/alternate provider to switch their system over if one goes out. It's called disaster recovery. The credit card company I worked for had an annual disaster recovery test. We had alternate sites setup and backup software ready to bring the company backup in case of a disaster. Our phone systems had trunk lines coming in from at least two different locations.
 

·
Registered
Joined
·
2,194 Posts
@Gentleman

I agree about not putting untested patches into service and I don't know what Rogers did. However, I do know that no matter how much testing you do, you can't catch everything and I used to do software testing at IBM in the late 90s. I also have decades of experience on hardware, telecom and networks that says the same thing. There are plenty of examples where someone didn't test for something, because that something never occurred to them. We'll just have to see what else they report.

BTW, I did my CCNA entirely on my own. I had classes for Netware CNA 3.x & CNE 4.x, and also did OS/2 Warp 4 on my own.
First, absolutely nothing or no one is perfect. No doubt.

I smiled at your statement because I was working on a large project in Calgary but one day I got asked to attend a meeting on a project I had never worked on and just the one meeting. Turned out it was on the IBM portion of the project(Iwas not involved in). At the meeting IBM decided they could "pass" on testing a portion of the work as it would be fine. Amongst a team I had never been involved with I tore into everyone about testing everything and no way IBM was going to skip that!!!!!! ;) That's when I realized why I had been asked to attend the meeting. No one was willing to stand up to the IBM crowd and someone knew I would! ;)
 

·
Registered
Joined
·
1,176 Posts
Someone in the IT dept needs to get the boot. The second time in two years a software update knocked out Rogers. Obviously no proper testing and no proper back out plan.
Lots of interesting comments, speculations, and "back seat driver" statements. Why should it be that someone in the IT department get fired rather than the CIO or CEO. No contingency or backout plan is infallible, and no testing plan can guarantee that problems will not occur when the same changes are made in a live production environment. Been there, done that.

More to the point, will they learn from this, and strengthen their process to prevent future outages. Every other competent service provider is looking closely at this event to make sure that this does not happen to them.
 

·
Registered
Joined
·
673 Posts
@JamesK

We are of the good vintage. My start in IT was during the era of the IBM system 370, token ring and 10base2. I've seen lots even making fibre optic patch cables and polishing the ends of SMA connectors.
Forced retirement now and that's likely a good thing.
 

·
Registered
Joined
·
10,648 Posts
What struck me was the scale of the outage. Why was every Rogers service and all Rogers customers' service disrupted? Was it because Rogers decided to update every device in the country at once? If so then disruption could have been avoided by a gradual rollout. That would have sucked for a few people but not the entire country. Was it because one or a few critical devices was affected? In that case then hot swap, mirrored backups should have been available. Yes, accidents happen but the scale on which this happened should have been preventable.

As to cost, that's easily paid for with the lost income this outage caused, both to Rogers and its customers. That such critical systems are so easily disrupted does not bode well if bad actors decided to disrupt systems, especially if they were a foreign country engaging in an attack.
 

·
Registered
Joined
·
8,939 Posts
@JamesK

We are of the good vintage. My start in IT was during the era of the IBM system 370, token ring and 10base2. I've seen lots even making fibre optic patch cables and polishing the ends of SMA connectors.
Forced retirement now and that's likely a good thing.
I started in telecom May 1, 1972 as a bench tech overhauling Teletype machines. I spent almost 23 years with CN Telecom/CNCP/Unitel and when I left, I was planning the installation of telecom equipment, both at 151 Front St. W. and at customer sites in downtown Toronto. My first Ethernet experience was with 10base5, connecting VAX 11/780 computers. When I was at IBM, the first time, I did 3rd level OS/2 support. We had token ring back then. The 2nd time I was at IBM was a few years later, on a 6 month contract, they'd switched to Ethernet.
 

·
Registered
Joined
·
10,648 Posts
From a customer viewpoint, it appears that Rogers has gone from a leading cable industry innovator to a 'me too' company that can lo longer reliably develop innovative products. There may be many reasons for that but the responsibility must ultimately rest with senior management. It took over a decade to get internet service to a point where it could fully meet the needs of customers but there appears to be major flaws remaining. Their wireless phone service has always had major flaws here and, unless things have changed recently, probably still does. Bell and Telus are now the industry leaders for internet, TV and wireless services but they are not without issues.
 

·
Registered
Joined
·
2 Posts
What kind of company has a network topology that takes out the whole network for internet, voice and television. And yes I realize it is all digital now so it is just data on their stream. But Rogers has just demonstrated that their infrastructure is neither robust, redundant and a probable single point of failure tripped their whole network. Thank goodness they haven't closed the Shaw merger yet and this demonstrates why it shouldn't be allowed. They do not have the competence to control this much of internet in Canada. I'm not even a Rogers subscriber but as a TPIA subscriber they took me out too. Who designs a network that falls like dominos, an issue in one centre should not take out everything from Vancouver to the maritimes.
i have everything with Rogers. It was not a great day. Our ignite TV is still stuttering, and last night was unwatchable. At the very least systems should run separately from each other where possible. Definitely going to move my cell phones at least to a different company.
 

·
Registered
Joined
·
309 Posts
Their so-called technical support, heavily advertised that they are in Toronto and not some foreign country, aren't very helpful.
I haven't had Yahoo mail since the outage and they said it's a Yahoo issue & to call them.
I said I was at my son's house & he has a VPN & when I pretended I was in the US, yahoo mail worked fine.
I also know Rogers email is "powered by Yahoo" so I'm guessing any Yahoo mail from a Canadian IP also goes through Rogers.
Tech support said that was not relevent & wouldn't escalate my issue.
Thanks for nothing.

Update
After a little troubleshooting I rebooted the modem (by unplugging) & magically I can now get Yahoo mail.
It also solved some other issues like certain websites not loading quickly.
A little upset at myself for not doing this "standard" procedure for Internet issues, but the outage just dominated my thought process.

.
 

·
Registered
Joined
·
309 Posts
A class-action lawsuit has indeed been filed by a law firm in Montreal.
It is asking for $400 per customer.
It has not yet been authorized by a judge but stay tuned.
 

·
Registered
Joined
·
4,388 Posts
They might get one or more days of service credit. What about the people who are out hundreds of dollars for lost wages or financial transactions? What about the small businesses who lost thousands of dollars due to lost transaction. What about the many millions of dollars lost by big businesses?
I see people saying this all the time. Most people have retail Internet + TV + Mobile or a combination of them which does not have any guaranteed uptime. If you want guaranteed uptime, you have to pay for that service. As an example where I work, we do have Rogers as our primary Internet and Telco provider and we pay for guaranteed uptime. Our 100MB Internet service is approx $1400/month Rogers did not meet that service level so there will be a financial penalty they have to pay. Our business also has a contract with Telus for backup Internet and Phone. The main issue that impacted us is we could not reach support to redirect our toll free service over to our Telus phone lines. What we ended up having to do is provide the local DID number to our backup phone system. Not ideal to say the least but we were operational throughout the outage.

It's not just Rogers that needs to be addressed here. Some of the critical services like banks, interac, police that were effected need to explain why they don't have carrier diversity. Any service that requires online presence should have at a minimum two different carriers.
 

·
Premium Member
Joined
·
9,502 Posts
Rogers class-action lawsuit filed in court after network outage
... The suit is seeking $400 for members who are Rogers customers affected by the network failure on July 8 and 9. It is also seeking compensation for Rogers sub-brand customers, like Fido Mobile and Chatr Mobile.

Arnaud Verdier of Quebec is named as the applicant in the legal filing. A Rogers customer since June 2020, he was "flabbergasted" that his carrier was only offering as compensation for the outage a credit equivalent to two days of service to customers, according to the application to authorize the class-action suit. ...
I'm flabbergasted that this guy thinks EVERY Rogers customer deserves $400. It's an undeserved gift to those who were minimally impacted by the outage, and insufficient compensation to those severely affected by it. A lawsuit demanding sliding-scale compensation would've made much more sense.
 

·
Registered
Joined
·
8,939 Posts
What I'm curious about is the people who claimed 911 wouldn't work. On Friday, my phone was showing no service and emergency calls only. If I had made an emergency call then, Rogers wouldn't have been involved. I wonder how people called 911. Did they just dial 911 as they might normally do? Or did they use the emergency call button, which does not require a valid account?
 
21 - 40 of 79 Posts
Top