Seems like a pretty big success to me. They had a contingency plan and it was implemented successfully. No cancelled flights and minimal delays - good job.
Presumably it helps that many (most?) travellers will be using the airline's apps these days, which provide info such as boarding times and gate updates.
In my experience, all the apps (airport, airline, travel apps, even Apple Wallet e-tickets) all have unreliable or heavily delayed gate information. Does anyone really rely on them?
The apps are touch-and-go. Some are fairly reliable (assuming wherever you are you have decent data connection), others are abyssmal, and none of them seem to be consistent in either regard.
I have found, however, if you sign up for SMS updates through most airlines they appear to be quite reliable, and sometimes beat the in-airport departure boards by several minutes, particularly for gate changes.
Going to the airline's site and looking up the flight status has often resulted in a better update on delays, gate changes, and etc. for me than even the airport's status displays.
I think it's evolving. For me it definitely used to be that way. I am a semi frequent Delta flyer and in the last year or so their app has become so reliable and useful that the only reason I need the airport monitors is that it is often a second or two faster to just look up at a big display rather that going through the clicks and swipes to get to the data in my hand.
A big improvement has been wifi availability and stability on the plane. Not so much because I can browse the web in flight, but because regardless of whether I get in flight wifi, the airline's app uses it for free. So now that wifi is reliably always there, I know what's up and where I'm going before I walk off the plane.
Info viewed directly in the Apple Wallet screen is delayed, for sure. That applies to all apps - the tickets/"cards" don't seem to refresh that often. But what you see inside the app should generally be up to date, although I guess not all apps are equal.
The US airline's apps tend to be very good, sending you real time notifications of gate changes and flight delays. On Delta, the app even took me directly to a rebooking function when it notified me of a flight delay during a layover in ATL. Very good UX.
I get Tripit Pro through work and I find that I usually get push notifications through the app before the flight information displays in the airport (!) have updated. YMMV.
I had a great experience with Lufthansa recently. Wasn't using their app, but received very punctual emails with delay and gate information, to the extent that the gate change email came at pretty much the same time as the announcement. Had a new flight, ticket, and gate automatically within about 10 minutes of missing my connecting flight as well.
Alaska Airlines app works really well in my experience. Sometimes too well, as the gate agent starts the boarding process and suddenly a crowd forms at the gate before they have even announced anything.
I built and maintain one such service. Hopefully we don't have delayed gate information, but we DO ultimately rely on airlines, and we sometimes get very confusing information.
This was a failure of a fibre connection to the internet and a cloud service.
Oddly enough, Gatwick's CIO was last week [1] telling journalists that he didn't like to rely on the cloud:
"The airport has used multi-tenanted public cloud in the past, but for that to work perfectly you need your telecom providers to work well, the path across internet to work, the hosting firm's telecoms to work, and their data centre support and monitoring not to drop. We have had examples where even in the public cloud using dedicated slices, you can suffer from noisy neighbours, and we just can't have that. Our operations are too important to us, so we need to keep it close. We use cloud for resilience, and we're very pro-cloud, but not for core services."
So it seems odd that departure boards were being fed from a cloud service. Maybe they hadn't been considered as a core service until now.
On the bright side at least the failure was limited to just the boards. The website, apps and check in desks all had the correct information.
reminds me of the waiting room at the auto repair shop I work at. As an engine mechanic, its vital to make sure customers have access to up-to-the-minute stats on their minivan (or so im told by my boss.) Eventually someone knocked a stepladder into the screen and broke it, so we switched to whiteboard and engine techs had to scribble in status updates in front of customers like some sort of weird lecture.
Techs would mark down "NG" for no go, OK for ok, and HOLD...these all make sense to us, but not to customers. Even worse, once an hour or two a tired tech would shuffle into the waiting room study the board for a minute, and slowly cross out the name of a person. The whiteboard worked, but that day our shop was indistinguishable from a mafia den.
I struggle a little to see how there is a single point of failure here (a fibre optic cable). I’d have thought the screens were powered by a device plugged into a LAN. Would be interested if anyone knows the architecture of such screens and why there is such a single point of failure in the whole airport.
These screens build information from multiple sources inc. airlines, ground control, and in some cases airport management. They're typically run off-site at a data center since that provides high resiliency and some data is sourced from outside the airport anyway.
If the airport has a dedicated fiber channel between their data center and airport itself and it was damaged, this could be the result. The question I'd ask is why they have no fallback strategies (e.g. cellular, VPN over a less secure internet channel, etc).
This might be an area where they can explore graceful degradation, meaning that gate information would continue to display (from airport ground control), but some flight times might be missing.
It's Gatwick, not Heathrow. Whiteboards are good enough. If every airport the size of Gatwick (and busier) implemented digital fallbacks, I wouldn't be surprised if the fallbacks took out the primary system more often than external causes. (I can’t remember the last time I used those screens over my phone. In a certain sense, the screens are now a backup.)
More precisely, if you consider the number of situations which (a) take down the primary system but (b) don't take out the airport, and try to cover those as cheaply and robustly as possible, for all but the world's busiest airports you come up with whiteboards. Localised power failure? Whiteboards. Surge? Whiteboards. Limited fire or flooding? Whiteboards. Fibre-optic cable gets nicked? Whiteboards or alternative digital system. Now compare the costs and risks of the digital system. Whiteboards are the work of a smart operator who probably saved their airport years of technical debt.
It sounds good in theory, but the practice is different.
“High availability” database measures are often less reliable than the normal, simpler approach.
Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.
Complexity is the enemy of robustness.
System failure is not just one simple category with known behaviours; different types of failures affect performance in different ways, and it’s hard to support all of them well. Just as exception-handling code is commonly the buggiest code out there, high-availability measures are commonly some of the most fragile parts out there, unintentionally causing more problems than they solve. They can be made robust, but it takes a lot of effort to do so, and frankly it’s seldom justifiable. Typically, simplicity is good. Definitely it’s easier to debug.
> Automated network failover equipment often causes more downtime than having multiple links and only failing over manually, or even depending on a single link.
Failing over manually sounds fine. If they can't swap over at all then that's a sign of fragility which will cause its own problems.
I don't have the data of course but it does seem better. Those alternative are pretty expensive to build and maintain, have to be tested regularly and are prone to failure. For a situation that has a very low probability of happening, whiteboards do seem more reasonable, that's a very lean approach!
The chance of a single fiber line getting cut is not "very low", it's something that a big entity should expect to happen. Much like a power outage; I'm sure they have very good plans for that case. An extra modem is cheap as dirt.
Not saying there was anything wrong with it. The people who came in during their off hours to update whiteboards for however many hours are the real heroes of this operation.
That said it was probably very expensive :P
Theres a lot of fawning over the humble whiteboard here, just wanted to emphasize that they were part of a larger scheme
It seems so strange that they wouldn’t have a redundant link (like the various options you suggested) or at least fall back to easily available public data (flight information sites) that would at least help some.
Some years ago, we lost access to our data center although it had triple network redundancy.
What happened was that three cables were leaving the DC leased by three different companies. But a mile down the road, all three different cable entered the same physical pipe. When the physical pipe was severed, all three network cables went dark.
We should wait until they announced details of what happened before we decide that we know what happened and offer solution ("If they only had redundant links")
If my memory serves, one requirement of high-tier datacenters is that they have redundant electric lines leaving the building from opposite sides and in opposite directions. I guess it must be the same for optic fibers.
I fully agree with you on your last sentence. One should wait for a bit more than a vague declaration reported in a newspaper before giving unsolicited advice.
How many people (even tech ops profesionals), these days, know how to tell the difference? Of those that do, how many don't just keep their mouth shut because their bosses don't want to hear it?
These details are just trusted to "the cloud", instead.
Of course, the airport itself isn't a datacenter of any tier, and that "last mile" can be the biggest redundancy challenge, anyway.
Yes, they leave different ends/sides of the building, but you don't have much control on where they have utility right of way and whether or not at some point all of your lines share the same right of way (which is likely).
I have heard this story before, with a twist that initially it was checked for, but the telcos changed it back some years down the road. I wonder what's a good way to audit this.
As someone above already mentioned — the whiteboards themselves also convey the message of “something is wrong, you may want to verify this info” very well.
While I agree that it's odd that there was no fallback, the whiteboards do immediately convey 'stuff is broken, you should check this board' pretty decently, as they would stand out like a sore thumb in the airport normally.
If there was a digital fallback, it would absolutely have to have something to tell commuters that the information might be stale (as someone frantically types up flight updates in a back room somewhere)
Perhaps they could have gone with a connectivity vendor with an extensive 4G network for backup if the (inexplicably) non-redundant fixed line was cut.
The problem is that it's the UK and mobile coverage isn't exactly world class here. You can get more consistent and reliable coverage towards the Russian border than you can around London.
The data on the screens themselves could be represented by nothing more than a simple CSV file. A dial-up modem could handle the bandwidth to keep that up to date.
[Gatwick has launched an app] for internal staff at the airport, such as shop assistants, baggage handlers and train station employees.
...
He says the need for the app stems from some passengers entering “panic mode” and expecting to get flight information from anyone who appears to work at the airport. They may approach an employee in a non-passenger-facing department who has no more information than they do.
...
“Passengers sometimes get more information from Twitter and Facebook feeds than from staff on the ground,” says Chacko.
“This is an attempt to bridge that communication gap and make everyone look smarter and more up to date.”
Distantly related but when I was in Europe decades ago I was at a train station where they had a massive split-flap display ( I want to say at a couple places ). I loved watching it update.
Aesthetically it was so much better than big super bright screens.
There's a company in the US that will make you one/rent you one, you can even control it via API. They sound really cool too.. I think I need to expense one for my office!
The board in Utrecht Centraal (the Netherlands) was taken down a few years ago, but you can still enjoy it in all its clackety goodness as https://itunes.apple.com/nl/app/blauwe-bord/id474162209 -- developed by the railroad company itself and showing real data off their open API.
South Station in Boston had one of these until maybe 5 years ago; sadly it's been replaced with a digital one. Though they do have a loudspeaker play a recording of the flapping sound whenever the display changes to alert travelers.
Yeh, whiteboard works but another more technical solution could be roll out a TV stand and hook up the TV to a laptop via HDMI and update schedule live through a shared doc (Google docs?). I mean, whatever works right...
That would require power sockets to not be a monumentally scarce resource inside an airport. Nobody is going to look for the TV rolled between a row of seats (that cause a dozen more people to have to stand or sit on the floor), facing a completely useless direction, to get flight updates. The airport isn't going to drag extension cords across the entire lounge without risking breaking somebody's neck.
They'll stare at the blank screens and wonder what's happening. A whiteboard in front of those screens, and other hot-spots, is far more considerate of how people behave inside a departure lounge.
Low-tech solutions to high-tech problems can be quite beautiful when executed well. It's a sign of good contingency planning because they anticipated no network availability, rather than betting on network redundancy.
I hope the few people who missed their flights were compensated in some way (refunds, seats on the next available flight...).
That requires them to have a surplus of portable TVs and laptops, which I don't see why they'd have (and would be much more expensive to buy as backup than the white boards)
As well as a whiteboard on standby, you also need 1 person per whiteboard to keep them updated. In an airport as big as gatwick, you probably can't get away with anything less than ~10 whiteboards (5 per terminal). Which means 10 people.
Not many companies have 10 spare people on site ready to act as emergency whiteboardmen.
There are 21000 people employed at Gatwick airport. There are going to be a lot more than ten people on standby at any given time, waiting to take on exactly the kinds of surprise situations which are business as usual in a busy airport.
If your displays are openable and have the right connector you could just use them. However, as someone else pointed out whiteboards work in a lot more situations.
At high-volume airports gates aren't assigned until the plane has arrived from it's inbound flight which is often <1hr from departure time.
Checkin generally opens 24 hours prior to departure (some airlines now allow >7 days prior) - hence for the vast majority of passengers the gate is unavailable as ticket issuance time.
Most airports and airlines let you get gate information fro their websites/apps as well though..
I think it depends on the airport & airline. Personally, in the US, I've had flights change terminals, not just gates. Terminal changes can be at least 20 mins on foot, even if you're fit and healthy.
Totally agree. In my experience, however, terminal changes occasionally happen but are unusual. Then, of course, there are places like Logan (BOS) where Terminal B has some sort of remote outpost- so even staying in the same terminal but changing gates can be a long haul to the connector because some of the gates are so far away
I've found that when the gate is printed on the ticket, it is rarely correct. Even more so for connecting flights. So in the end, I still have to look at screens until I am actually at the gate.
I've found this too. The terminal usually stays the same because they are contracted to specific airlines, but the gate is fluid. For example, my wife is flying home today and her arrival gate was only announced a couple of hours ago (she's taking off just about now).
There it annoys me the most. I always know which section of the airport my flight connection is and it’s a 20 minute walk. I rather go there now and sit down with my laptop and work undisturbed.
Went volunteering at a speed skating event once. One task I was asked to help with was mark the order of the skaters at the end of the race. I asked why that is and they explained that if all systems fail (1. Device on the skater's ankle, 2. cameras recording the race), then this ordering we note is what they will fall back on.
It seems like a lot of work people do is individual and idle when other systems take responsibility of many issues in a process. I'm glad that people worked closer together as a result of this failure!
I'm glad people worked closer together here, not that there was a failure!
Imagine having to update the whiteboard in real time though? Must be a right PITA, unless there is minimal changes going on with the flights (lack of delays or gate changes).
This is the opposite of a garden path sentence: the first verb really is part of the sentence's main clause - i.e. the meaning of the sentence is "Whiteboards [are] used..." instead of "Whiteboards...fail".
Not everyone has a phone. Not everyone has that app (whatever it is... the airport, the airline, the travel agency), using an app is always harder than looking at the real-time displays especially if your hands are full of luggage.
The airport's whiteboard backup plan is interesting but my first thought was also that all of this info is available online or through the airline apps.
In my experience, changes often show up in Kayak ahead of the airport dashboards. I check apps or notifications before the official dashboards now.
