Wow! I worked on EODSIS in 93-96. We estimated 16 petabytes, at the time it would be one of the worlds largest databases. We changed horses midstream moving our user interfaces from X-windows Motif to WWW. And built a very early Oracle DB accessible via WWW. There was no cloud then except missions studying atmospheric water vapor.
When this was originally designed there were to be several (6-7) DAACs - Distributed Active Archive Centers (https://earthdata.nasa.gov/eosdis/daacs) to store data near where it was needed or captured. Now they have 12 and are storing on AWS. Amazon didn't exist when this was originally built.
1. Using the AWS cost calculator is pointless, naturally an entity the size of NASA would get heavily discounted rates.
2. As data volume grows, the complexities of working with that data expands. NASA appears to be embracing cloud computing by embracing a paradigm where scientists push computation to where the data rests rather than downloading data [1], [2], [3], thereby paying egress on only the higher order data products.
3. The report notes that NASA has tooling to rate limit and throttle access to data. This, in itself, proves that NASA didn't "[forget] about eye-watering cloudy egress costs before lift-off".
People may scream about vendor lock in, which is a fair complaint; but acting like NASA just didn't think about egress is misleading.
NASA is ultimately a science institution, I think diverting effort away from infrastructure management and towards studying data is likely a wise decision.
Datapoint: When $company hit very high six figures (closing in on seven) in monthly spend I found AWS was incredibly willing to cut our egress rates, often by a significant amount.
This was explained by our account team: bandwidth has some of the best margins for AWS but they’re willing to sacrifice that for their enterprise customers (read: suck us in closer to non-commodity services)
$75k/mo is tiny in the enterprise world. At Oracle they’d give a 22 year old fresh out of school ~30 accounts that size, for reference. I worked on a team of 9ish on a ~$5MM/mo account. (Not cloud, but a comparable business unit)
The big players have market caps measured in billions, so there aren’t a huge amount of them. IMO cloud is weird since for most products/services you can go buy from a smaller company to get better customer service, but that’s obviously not the case for aws/etc.
I've been on teams spending half of that, and managed to get great discounts.
My question whenever I hear that people didn't is who did they ask? AWS doesn't just jump in to give people free service- but if you reach out to them and tell them you need it they tend to work with companies.
>NASA is ultimately a science institution, I think diverting effort away from infrastructure management and towards studying data is likely a wise decision.
Indeed. I am glad to see them leveraging the power of an already proven infrastructure provider rather than spending X billions of dollars trying to build and maintain their own.
> the power of an already proven infrastructure provider
Every major cloud provider is using Linux network drivers written by NASA employees.
In 1994 the NASA Beowulf project pioneered the idea of clustering together cheap commodity hardware to replace mainframes (this concept was later used to bootstrap hardware for Google)
NASA helped start the OpenStack project which powers a number of cloud providers.
Heck, NASA helped invent the GRiD Compass, the first laptop computer.
No, that is just ridiculous. NASA is more than capable of running their own server infrastructure. They've got expertise, they've got DC's and they don't need 99.999% uptime for most of their services. Cloud providers can turn out to be insanely expensive. I am not against cloud - mostly I would recommend it for businesses but when reaching a certain size you have to consider doing your own cloud infrastructure.
Yeah. It seems like everyone is hopping aboard that bandwagon and doesn't remember a world before 2009. For $1M, you can get yourself a very beefy server farm.
AWS accounts still take management and a team of people that need to maintain a whole lot of different aspects of it, so you're not really saving on headcount. You're just moving that capex to opex.
It's important to be flexible enough to be able to deploy onto a cloud provider if the situation demands (e.g., new client demands infrastructure run in $FOREIGN_REGION_X where you don't already have a DC), but everyone's insistence on going 1000% AWS is absurd and IMO totally unjustifiable.
I agree with you.... but no one wants the responsibility of securing the servers and keeping the hardware up to date. I assume NASA has an abundance of outdated hardware and no one to sell these outdated systems to. My fear is we will lose this ability to create our own servers in the future. My degree program had nothing dealing with cloud, I would tell the professors that IBM blumix and M$oft Azure was the future. My outlook toward the future was close, who knew the book store would be the biggest cloud provider. Nearly every company I worked for had a big initiative to go to the cloud. One infrastructure guy told me the final decision came when they realized they had no way to train the next generation, cloud upfront cost is less risky, and its nice to have someone else to point fingers toward when things do not work as expected.
Sure, they are certainly capable of doing it themselves, but why should they?
For what AWS provides, the DIY approach would be insanely expensive and wasteful. Not to mention, it would take years to build a basic MVP. They'd have to scope out the project, hire people just to design it, and then pay for several contractors just to stand up the first iteration of a working system (which will still not compare to AWS in terms of resiliency, redundancy, and accessibility), and then maintain it....forever. Also, many people already know how to interact with AWS. NASA would also need to design and maintain user-access methods to the data, on top of just plugging in thousands of hard drives and making them all work nicely together.
Why reinvent the wheel when there is a perfectly good wheel manufacturer that has already proven extremely successful at what they do?
>NASA is ultimately a science institution, I think diverting effort away from infrastructure management and towards studying data is likely a wise decision.
True, but once you're a certain scale, outsourcing everything just because it's not your competency isn't a good excuse. You can afford to hire enough people for it to become your competency.
Probably there is even good competition between the cloud providers, because hosting the data means that you can sell a lot of compute time to all the users of the data. NASA choosing for AWS means that any IO intensive analysis on that data will run faster/better/cheaper on AWS.
Yeah, this looks like a FUD hit job, possibly by entities made obsolete by a move to AWS.
There are just too many solutions to egress optimization to mention (CDN edge caching, rate limit, throttling, tiered discounts, multi-year agreements).
No gov procurement deal at this scale gets sticker shock from retail prices.
Totally and utterly not a FUD hit job. Just a reporter finding a document that told a story. I am that reporter. And FWIW when people push a dirty story that's clearly in their interests to have in print, I either don't write it because I won't be their cats-paw, or I mention it in the story. This was in plain sight, but I guess hadn't been read by anyone who understands cloud.
Using the AWS cost calculator is pointless, naturally an entity the size of NASA would get heavily discounted rates.
I strongly doubt this.
Amazon seems to work heavily on the principle that they charge their cost + a small margin. Which means that they can't heavily discount without going below their actual costs.
> “However, when end users download data from Earthdata Cloud, the agency, not the user, will be charged every time data is egressed.
Not necessarily, depending on how the users access the data. If users access the data through their own AWS accounts, NASA could leverage S3's "Requester Pays" feature [1], to let the user pay for downloading the data.
I immediately thought about this as well, however I seem to recall reading somewhere (and I could be entirely wrong here) that NASA has a requirement to give away freely their science data.
If there's a marginal cost for each copy of the data that's transferred to a user, I don't think asking the user to cover that cost conflicts with a requirement to "give away the data".
(If they distributed their science data in printed form, surely they'd be allowed to charge people for the cost of printing & mailing the paper copies; that's quite different from charging for the data itself.)
Why the downvotes? This isn't uncommon or unreasonable if you're downloading TB's of data. Also the data would be freely redistributable if someone took the data and put up a torrent. Still I'd rather see NASA host their own data. Put up an FTP server, torrent server and save a lot of money on hosting fees.
While proxying through a torrent system is a good idea. I doubt it would get well seeded outside a few popular datasets- the agency would end up the sole seeder of the long tail.
I’m willing to bet NASA saves a ton of money by going to a cloud provider- US government storage setups are insanely expensive. I remember a project I was on got a quote of over $10,000/TB in 2014, and there is no way egress is actually free right now- they are paying for a government regulation compliant internet connection one way or another.
I do worry about vendor lock in to a degree, but I’m confident the agency and tax payers would save money going to any major cloud provider.
Sounds like there is a bigger story there and it's probably a managed SAN.
I've operated pretty significant government shared infrastructures like this in the past... we were offering fast, flash-cached disk in 2010 for about $5,000/TB. $10k/TB is not unreasonable for highly available Tier-1 storage for something like SAP, especially in that era where you couldn't use all flash in most case.
Today, cost structures can be very different. You can land high-iop storage for a fraction of the cost without the overhead of a big SAN. If you need capacity focused storage, that is also much cheaper.
An agency like NASA gets hosed on services, and cloud is no different. AWS is probably a net savings for operational workloads whose characteristics are known. Backup is a no-brainer. But for a high-volume, operationally highly variable thing like a public archive of data, AWS a square peg in a round hole because of the metered access.
I’m sure that $10k/terabyte quote was complete overkill for what we needed- but that’s what the stove piped storage org was offering, and it killed the project we were working on.
I wasn't really sure what they pitched us technically, but your pitch sounds reasonable. It was also complete overkill- we were hosting
read only static images (map tiles). Azure and AWS were less than $300/TB/Year at the time, and their triple replication was more than what we needed availability wise.
Because the storage group refused to sign off on a cheaper solution with lower specs (I don't know why) and acquisitions in the government is a mess so going outside would have tied up one of our primary constraints (the tech lead) more than it was worth.
The overall system ended up with worse capabilities than it should have had, but it did ship.
Wow! That's good to know, if a bit disheartening. I guess I was thinking costs for small startup costs with some cheap-ish linux raid setups and likely massive fiber taps NASA must surely already have. Not government/big business costs.
You'd be buying something like an EMC vMax that can sustain 1M+ IOPS on lots of 15K spinning drives, with caching tiers on crazy expensive flash.
To support that, you need a fibre channel network layer and a bunch of FTEs to attend to it. Usually compliance requirements require segmentation of roles, which increases cost. If you're a federal government entity, those FTEs are most likely contractors billed out at $125-300/hr. Figure $3-5M/year on labor costs alone, although that may be divided out over multiple systems.
This happens in commercial business too. I had a buddy who was making about $150k in NYC to zone luns on a SAN. Basically he kept a spreadsheet and updated a specific configuration setting 2-3x a day and spent about 60-90 minutes/day doing that. The rest was waiting or studying for his MBA.
It's pretty wacky to compare S3 to this type of storage.
At a technical level yes, it’s wacky. At a “this is what government departments actually do” level, it’s perfectly reasonable. I’m sure NASAs current system is actually pretty efficient as the us government goes, but having spent a career running into the sort of institutional pathologies that lead to an interdepartmental quote for $10k/terabyte, I’m willing to bet AWS is very competitive.
Records departments always charge for copies, and that is the use I thought of immediately when I learned of Requester Pays. I’d be surprised if NASA couldn’t use it.
> If there's a marginal cost for each copy of the data that's transferred to a user, I don't think asking the user to cover that cost conflicts with a requirement to "give away the data".
Charging the user for data, even if it is on a marginal cost basis, conflicts with a mandate to give data away freely. Because “at the marginal cost of delivery” is not “free”.
(It's true that it is common for mandates to specify something like at marginal cost of delivery rather than free—sunshine laws providing copies of public records often work that way—but that's not the applicable mandate here; in fact, since without the separate mandate here the data would be available on a marginal cost basis under FOIA, the main reason for a separate mandate is to negate that cost.)
Also, public access can mean that once someone gets a copy of the data they can host it for free as well. It's not as if it's under a commercial license.
While the data is free, the cost of getting the data to you can be charged. Originally, it was to cover the expense of someone pulling the data, making copies, and then mailing that data out to you. If it was photographic, you'd be charged for the prints. I'd see using Requester Pays in the same vein. They are not charging you for the data, but any fees incurred to obtain the data would be at your expense.
Isn't requestor pays just like I pay for gas to drive to my local library, when I can't bike because I want to borrow so many books, but the books are free to loan.
It's required to be public domain. IMO it's comparable to FOIA requests still requiring the requester to attach a stamp to the envelope their request goes in. Or at most, include a self-addressed stamped envelope too.
Requiring you to pay S3 is little different than requiring you to have Internet access, and thus pay whichever company includes you in THAT monopoly, IMO.
Imagine for a moment that in order to access NASA data sets you had to have a Fastmail email account. Gmail won't work, Outlook won't work, it has to be Fastmail alone.
That would be very objectionable (as much as I adore Fastmail).
Ability to pay one specific cloud provider should not be a gate for public domain government data.
I don't think this analogy works. For Fastmail, there is a cost regardless of whether you want to access government data. You have to pay for the account itself. For most cloud providers, there is zero cost for having an account. Even if they hosted this themselves, they could just as likely charge for data transfer costs...and get to choose how to collect that. They could choose PayPal and you have to create an account. Or they take credit cards...and you must have a card belonging to one of the networks they support. The barrier to entry doesn't change regardless of how many cloud providers there are, all it does is increase infrastructure costs unnecessarily.
The alternative here, though, to get comparable distribution / durability, etc. by spending way more of the public's money upfront regardless of who wanted it. I get the purist / idealistic argument here, but it feels a bit like cutting off one's nose to spite their face.
I'm not an expert, but most government agencies are allowed to charge reasonable fees for access to their data. I don't know if this qualifies, but it at least seems like a possibility, especially if it's transparently just passing along their costs in the form of AWS' own cost structure
I wonder if there is a problem with this because it requires you to have an Amazon account and such to do it. There is now a much higher barrier of entry for random people to access small amounts of data. And no longer have direct http links. You have to use the CLI / SDKs once requester pays is on there.
Because it allows the agency to escape from its bad design problems by pushing the (huge) cost onto its clients -- and those clients are other parts of the US Govt or funded by the US Govt.
You're asserting the design is flawed when that's in dispute.
It's useful for those agencies' budgets to reflect a portion of the cost of performing that research.
The USG needs insight into what taxpayer dollars are being spent on. Lawmakers have to explain to constituents why that money is being spent.
NASA is the first tier of information, collecting the data. Its budget ought to reflect that cost.
The consuming agencies are the second tier, processing that information. Their budgets reflect the cost of gathering their information and of processing it.
NASA doesn't know which information will be useful, so it's not helpful for them to pay the cost of egress. We want them to collect as much as possible.
It's much like a music store, 90% of their sales come from the top 10, but there's a lot of value in hosting obscure stuff.
If they have to pay to store it all rather than pay for egress, they'd have to justify the cost storing data that they can only say "it might be useful some time."
Having the agencies that are working with the data pay for the egress, they can justify the cost by showing the specific work they do.
I'm a huge fan of requester pays, and I frankly don't understand why we haven't switched more of the internet to it.
I'm also a liberal, so then I also think government should give everyone a monthly quota of internet usage allowance. Universal Basic Internet Income, or something.
I'm not saying this won't be a financial cluster - it likely will cost many times more than planned - but the headline here is just a flat-out lie.
TFA says:
"a March audit report [PDF] from NASA's Inspector General noticed EOSDIS hadn’t properly modeled what data egress charges would do to its cloudy plan."
'Hadn't properly modeled' is very different from 'forgot about'. And if you actually read the linked report, it says things like:
"ESDIS officials said they plan to educate end users on accessing data stored in the cloud, including providing tools to enable them to process the data in the cloud to avoid egress charges."
and
"To mitigate the challenges associated with potential high egress costs when end-users access data, ESDIS plans to monitor such access and “throttle” back access to the data"
Neither of those statements would be in the audit if the entire topic had been a surprise.
"In addition, ESDIS has yet to determine which data sets will transition to the cloud nor has it developed cost models with the benefit
of operational experience and metrics for usage and egress."
YOU ARE NOT AFRAID?
'Not yet. But, er...which way to the egress, please?'
There was a pause. Then Death said, in a puzzled voice: ISN'T THAT A FEMALE EAGLE?
I've been reading A Hat Full of Sky to my daughter these days, and there's a running joke that "supposedly intelligent people" don't know the meaning of the word "egress", mixing it up with things like egret, ogress or eagles.
It's The Register, people. Don't take it seriously. It's practically The Onion of the IT industry, especially the comments sections.
I've written two articles for them and the comments are a joke. They're all anti-Cloud, anti-progressive. Try selling them Kubernetes has a solution to their problems: they'll think you've come to steal their children. I know, I've tried.
In short: this never happened. NASA didn't forget anything. It does, however, make for a great eye catching headline!
Sorry to be bitter about this, but publications like The Register serve little purpose these days. It caters to a specific kind of IT personality that can't let go of their physical tin and they think public Cloud has no place or use at all. Again I know, I've tried convincing these people of such things.
Smartest comment so far in the thread. The issue of cloud egress has been known and worked at NASA for a decade now, and the article treats it like an OMG moment.
Historically, data have been stored and processed on-premise but NASA has been migrating data and processing to the cloud where it makes sense. For instance, it makes a lot of sense to burst out to the cloud for near-real-time processing during and just after natural disasters like earthquakes and forest fires.
The large missions they mention (SWOT, NISAR - big radars in Earth orbit) are drivers of the shift of more processing + data to the cloud, because they will generate an unprecedented amount of data. They are pathfinders. By percentage, very little of that data will ever egress - it's low-level and uncalibrated - so a cached strategy could be valuable.
Here are some slides giving background on the SWOT/NISAR data system. They are from 2017, so more has happened in the meantime, but they touch on some of these issues:
I know. And I'm saying if that was the rate they've historically added data to their dataset, it would've taken them 200,000+ years to get here. Which is why 100GB/mo is virtually nothing for NASA -- it doesn't match with their historical throughput.
That's the default value on the calculator, but all that does is prorate the storage demand over the period. Backblaze is entirely usage-based (speaking from experience; I'm a customer).
They've owed me money (just a few $) for several years now, in my Amazon Seller account.
And they've promised to pay me, something like 20 odd times. With a specific date each time.
Which is never paid.
And ever time I ask what happened, the customer service person says they'll look into it, and never get back to me.
Most recently, they've sent an email saying they're closing my account due to lack of activity. No word on what'll happen to the funds, my expectation is they'll just steal the money for themselves.
"Lack of activity"... yeah, no kidding.. People tend not to use a service when the other party is obviously full of shit and repeatedly lies. :(
As for "AWS never forgets", sure. That goes both ways.
How much would a launch pad that will be used four times normally cost for what they're planning to launch? Without knowing that I can't say if they overpaid 10x, 2x, got it exactly right or got an amazing bargain.
In 1965, the Vertical Assembly Building, which was at that time the largest enclosed volume in the world, cost $117M (on a $23.5M original construction contract). That would be about a billion dollars in 2020, but it was completed in 3 years and was used to stack 13 Saturn Vs. It was later used for the 100+ Shuttle missions as well, but there were additional costs to modify the building for this purpose. The VAB is still planned for use for future missions.
I picked the VAB because it's current dollar cost was roughly a billion dollars.
Total cost for constructing Launch Complex 39, which includes the VAB and the crawler-transporter launchers was estimated 1t $500M in 1962 for 2 pads. A total of 153 launches have occurred from LC-39. This number is greater than 4.
I assume the data accessed is a heavily skewed pareto distribution.
Given that, it's maybe still cheaper to build their own serving / caching layer in front to save egress costs than to have constructed the whole storage solution themselves.
Putting a caching layer in front of AWS is often very cost effective even without much skew in the access pattern. It tends to take a very low hit rate before it pays for itself.
This surely was entirely known to AWS, where they were rubbing their hands at the fact that every user of this data has to process it using EC2 on site.
They could use direct connect, from each of their data centres, essentially turning AWS into a giant NAS. However this gives up the idea of using AWS compute to provide value added analysis.
Cue the cloud apologists that “it’s better to use the cloud than to build and manage your own infra”.
This is why you build and run your own storage, similar to Backblaze (who is almost entirely bootstrapped except for one reasonable round of investment).
To cloud or not to cloud is the same as any outsourcing decision.
For many operations, you may get to a point where it makes sense to build your own cloud.
If you're a seller, you might also get to a point where you want to sell goods directly.
It partly depends on your core expertise, meaning, is this part of how your business creates value? If NASA doesn't want be a datacenter provider, they should continue to outsource it.
It also depends on whether their business model aligns with yours. AWS's egress rules specifically work when you are getting revenue from the data being downloaded. If you're selling software or other media, and you can factor the cost of downloads into the price of it, pay-for-egress is very sustainable.
Other models like pay-for-capacity don't align as well if you want to maintain a large library of media and people are attracted by the variety, but only download the popular stuff.
For NASA, pay-for-egress may be entirely justified if their budget is based on usage of the data. Or if they can simply use "requester pays" to mitigate the cost.
Cue the cloud detractors that "a failure to do due diligence (in this case: 15 minutes on the pricing calculator) on your computing platform should be held against the whole platform".
Snark aside, it entirely depends on what you're doing. AWS probably has better engineers, better processes, and more of them than your company.
Due diligence only somewhat mitigates the damage done by having a generation of engineers who believe going straight to AWS or another expensive cloud provider is the first and or best course of action, when you have engineers scoff at building a cheaper, more efficient solution better fit for purpose. Backblaze proves it can be done, and I argue they are just as competent, if not more, than Amazon. They’ve provided a similar object storage system as S3 at a drastically lower cost.
In most scenarios, it’s not my money, and I don’t care if it’s not my money. In this case, as a taxpayer, it’s my money (our money to be specific) and I care. I intend to contact my representatives about this failure, and have already fired off a FOIA request for AWS NASA contract details.
None of which will really help you, since AWS priority is AWS, not the uptime of your business. And no number of those better engineers or processes have prevented downtime and service interruptions on AWS.
Better run your own Internet, after all, you care more about connectivity to your friends than your ISP does!
Dogmatism is passé. There are good uses for cloud, and good times for on-premise, depending on what you need, what your skillsets are as an organization, the kinds of workloads and length of time required for that workload.
AWS and others have absolutely outstanding amounts of infrastructure and tooling. Their reliability is off the charts in the past few years, and (once it actually gets figured out by your engineers) the cloud concept of IAM is incredibly secure.
There are pitfalls - cost, up-front complexity and several other things - but I no longer rag on "the cloud".
Amazon has outages all the time, hidden on their status board with a green triangle, and you still lose S3 objects once you’re operating at a large enough scale.
A quick google search for “amazon outages” lists the numerous extended outages they’ve experienced.
How many of those outages were multi-region and would have taken down a properly distributed application? How many outages and instances of lost data would the average enterprise, likely without their own datacenters, redundant power, hardware staff, etc have taken in the same period?
Most applications will never be architected to be “properly distributed” because of cost. Many popular web properties (Reddit) still have outages on AWS even when architected properly. Netflix still distributes content from their own CDN with their OpenConnect appliances, and only uses AWS for non streaming use cases (jedberg will correct me on both Netflix and Reddit points if I'm missing something and comes across this comment).
If my app is architected for reliability, I’ll run it on bare metal and keep the costs savings. Why pay twice by building it for cloud durability and running it on expensive cloud resources? Clearly the AWS marketing is working (“you’re just building it wrong”).
We’ll see what happens when CFOs take the reins from CTOs and CIOs and start putting cost controls in place during this recession (“why exactly are we paying so much in opex when this could be capex we can depreciate?”).
Ok, so we replace a lot of opex with a little capex and a lot more opex. You only need devops types if your business runs on a cloud provider, now you need to employ facilities, sysadmins, security, etc. It's not just the cost of the hardware we're talking about, your labor budget will necessarily increase as well.
On a tangent from the sibling comments (which are spot on), colocation does exist. They handle the network drops, power, security, cooling, and you just have to ship them servers. Before AWS, this is how most businesses ran (including Amazon).
Few businesses ever get to the point where they need to run their own datacenter. And when they do, the costs would be roughly even or lower to AWS due AWS' markup (for handling those DC-related things for you, plus profit).
Devops types are sysadmins that cost more for mostly the same skillset (you know cloud primitives, you know infra as code, you know some python/bash or powershell depending on the underlying OS). Facilities, security, etc are usually covered by your hardware hosting provider, or colocation provider. Still a lower cost than cloud. You are still paying similar labor costs regardless if you're in the cloud or have your own metal.
Disclaimer: Previously a devops/infra guy, before that ops/networking/sysadmin, built out colo facilities/datacenters/hosting companies before cloud. Have done a lot of cost models for storage and compute, still do on the side.
So who takes care of the non-development tasks that AWS (or any cloud provider, really) is handling on the backend? Schlepping the hardware around, swapping failing drives, hardware monitoring, actually speccing out and running a datacenter, physical security, and so forth?
It's generally not the same people who are going to be at their computers running awscli (or if it is, now we get to figure in how much time they're spending on tasks that are not their primary job and how many extra of them we get to hire to maintain the same velocity, not to mention the occasional bit of firefighting you get to do when you manage your own infra)
I thought the ultimate argument was that if you're big enough AWS will make you a deal. But maybe now AWS is just so big and already growing so fast, they don't want to make exceptions and lower their profitability.
"At least NASA seems to have bagged a good deal from AWS: The Register used Amazon’s cloudy cost calculator to tot up the cost of storing 247PB in the cloud giant’s S3 service. The promised pay-as-you-go price for us on the street was a staggering $5,439,526.92 per month, not taking into account the free tier discount of 12 cents. The audit, meanwhile, suggests an increased cloud spend of around $30m a year by 2025, on top of NASA’s $65m-per-year deal with AWS."
$5.4m/mo * 12 mo/yr = $65m/yr. My guess is the "$65m/year deal with AWS" is actually the S3 cost and the extra $30m/year of 'increased cloud spend' is the egress costs found by the audit. Otherwise it's a coincidence of the numbers.
> You don't need to be a rocket scientist to learn about and understand data egress costs. Which left The Register wondering how an agency capable of sending stuff into orbit or making marvelously long-lived Mars rovers could also make such a dumb mistake.
I used to work very closely with this department at NASA. Without saying too much, the short answer is "tenured government employees more concerned about job security than the success of the project" is how an agency could make such dumb mistakes.
Using AWS for this type of use case is dumb for an org as large as NASA, if cost savings is a goal. It's cheaper to just land capacity at a datacenter.
I guess they have additional legal constraints that don’t allow them to just “land space” here or there - the vendor must probably be security-vetted, compliant to a hundred government-produced checklists, and willing to go through extra-long sales and support cycles. It will inevitably push up prices significantly.
In fact, I can imagine ops-teams at Nasa licking their lips at the idea of doing away with a lot of that bureaucracy once they switch to AWS... note how the report mentions that some of the controllers are actual sponsors of the move: it’s obviously a conflict of interest, but it might well arise when the org as a whole is a bit too happy to steer away from a suboptimal situation.
This said, AWS will rob them blind, simply because they can. Like all outsourcers (which is effectively what they are), they get in with the simplicity argument, then boil that frog up with extra charges. It’s good that somebody pointed out one of those charges, but I doubt anything will change substantially- Amazon will probably cut them a discount and that will be it. And once you’re invested in a cloud env to the tune of hundreds of petabytes, you’ll likely not switch away for decades.
That implies a level of dishonesty or nontransparency that AWS doesn't have. Their pricing is disclosed, up front, and they offer a calculator to model your costs out. Knowing how much data egress you're going to have is not some arcane art, NASA just plain forgot to do it.
It may be complicated, but so is any workload at this size. Figuring the cost is part of due dilligence, and they've made it as straightforward as possible.
> That implies a level of dishonesty or nontransparency that AWS doesn't have.
Have you ever been part to an enterprise-level sales cycle? Things like the official calculator are waved away, since the customer is on a special deal, so "of course is not as much as that!". The customer asks for a quote with a certain degree of detail, the vendor provides an answer with the degree of accuracy required to get them in the door. If it turns out after a year that the customer ended up paying 2x, well, too bad - clearly they must have had higher requirements than forecasted! "Did you record all your traffic? No? Well, we did, and the result is this bill, sorry. Alright, alright, I hear your complaint, I tell you what - I'll give you a big discount on your next order, what about that?" Rinse, repeat. This is not dishonesty and I'm not alleging malfeasance or anything like that, it's just how that world works in my experience.
In order to figure out the real cost of outsourcing, you need an adversarial attitude that most shops simply lack, because they've fundamentally made the choice to abandon the previous solution even before they've entered the sales cycle. This is particularly clear in a case where some controller is also part of the group promoting the switch. It's surprising it was flagged up, there must be a competing group somewhere that is desperately trying to fight on - maybe some Oracle-friendly "japanese in the jungle" or something. Or maybe bureaucratic procedures to safeguard the institution are actually working as they should, for once, but that would be pretty exceptional in itself.
All of the cloud vendors de-empathize network egress costs. It's similar to products that depend on Microsoft licensing who will always omit those types of costs. (Oh, so you needed to spend another $500k in SQL Server Enterprise?)
Many organizations lack the operational metrics to allow them to effectively measure their egress needs. And AWS/GCP/MS salesmen arent in the business of slowing down deals with awkward questions.
This is especially true where an org like NASA probably contracts out things like network services. Going from a model where you make fixed capital investments to paying for the byte is difficult to measure.
Here's the official pricing calculator[1] - note that ingress and egress costs are included in all relevant services. Also note that for something like S3 (which is probably what the article mentions the "earthdata cloud" is based on), the pricing details are right there on the description page[2].
There is no evidence of any malfeasance by AWS here, just lots of casting aspersions. What specifically do you want that was not provided?
This article is misleading. The entire point is to not move data out of the cloud. Instead bring your computing (analysis, visualization) to the data and pay for compute cycles on AWS. If your workflows are short/bursty, you will come out ahead. Moreover, you will be able to do big data-style computations that you cannot do in a local computing environment. This is bad journalism, IMO.
If you are facing similar problems you should know traffic via Cloudflare from B2 is free. I am not 100% CF would be happy if NASA picked the CF free tier but probably their quote would be magnitudes lower than Amazon's.
Exactly. Move your computation to the data instead of the other way around. At that point, there are many ways to keep costs down such as using spot instances and tearing down VMs when your analysis is over.
Wasabi does not charge for egress but our pricing model is not suitable for use cases involving the hosting of videos in a manner where the ratio of egress downloads exceeds the amount of storage.
This is exactly why the costs are set up that way. The first time I saw AWS pricing I chuckled and thought "roach motel." Data goes in but it doesn't come out. Its one of many soft lock in mechanisms cloud hosts use.
just build your own storage and save an incredible amount.
It's hard you might think, but it's not. croit.io provides all you need to deploy a scalable cluster even on multiple geographic regions.
Price for 1 PB sized cluster including everything from rack to hardware to license to labor for below 3€/TB/Month or at the Amazon Glacier price tag but with the S3-IA access.
Do you want to add your contact details to your post so NASA can get in touch or what is going on here. Add a little disclaimer that you work for/are croit.io so people can instantaneously see why you would argue for the space agency of the U.S. to run their own data storage.
Seems like a poor choice. If they're getting an incredible deal with AWS, then fine, but I would be utterly shocked if most seasoned and competent IT professionals couldn't design and build a multi-region storage array for far less than Amazon will charge them.
Torrents are only helpful when there's a large number of people who download the data and are willing to share it. There's not a large userbase for the vast majority of NASA data. It wouldn't be distributed in any meaningful way.
Your numbers are way off, as you didn't account for redundancy of the drives (any failure or bit flips of 1 of those 2,470 drives will cause corruption of likely the entire data set).
> Network cards, bandwidth, electricity cost > I can't guess.
This is where a huge amount of cost is.
> And that won't be recurring cost.
Maintenance, humans, cooling, drive replacements, property, building, land tax, payroll tax are all recurring costs.
> Your numbers are way off, as you didn't account for redundancy of the drives (any failure or bit flips of 1 of those 2,470 drives will cause corruption of likely the entire data set).
Let take another setup of same count as backup. Then another setup as back up of back up. ~150K
> This is where a huge amount of cost is.
Maintenance, humans, cooling, drive replacements cost > can't be greater than first time set up cost.
> property, building, land tax, payroll tax
Nasa runs on Government budget, I am sure they can claim some tax break there.
The point I am trying to make is, it may be cheaper to do in-house with the level of engineering talent they have.
Have 3x sites around the US the build the pods, each new pod gets preloaded with a smattering of rarely requested and low replication count objects (as a redundant backup). Then shipped to the site where it will be used. Local writes go directly to pods which are then kept in sync with the rest of the cluster.
edit, from the TFA
```
And to put a cherry on top, the report found the project's organizers didn't consult widely enough, didn't follow NIST data integrity standards, and didn't look for savings properly during internal reviews, in part because half of the review team worked on the project itself.
```
You realize that the entire openstack project came from the opensourcing of NASAs opennebula project, right? They've got one of the biggest infiniband networks in the world underpinning it.
In addition to saving money, they will also make the US more resilient by helping avoid a concentration of expertise and an infrastructure mono-culture.
I suspect that ideas like this will become more popular as the US asks itself "what happened to our resilience?"
