I had wrongly assumed chain of custody manifests are a thing in Snowball and AWS Import/Export. We just discussed on here about ECC memory where the consensus was ECC RAM even in laptops is good (especially at 32GB+), yet somehow mass storage dealing with orders of magnitude greater volume of data is given a pass on transits. Thanks for sharing your experience, good to know I have to preserve our manifests discipline outside our data lakes' borders.
Also interesting to find out AWS uses their customers' resources for temporary storage in the transit process, instead of elastically using process-only-bound ephemeral resources outside customer space in a more cloud-native fashion. Temporary consumption in the customers' resource space in a solution pattern gives me nightmare scenarios of stray code that scribbles the temporary objects into customer-owned data, or accidentally dropped into the wrong location and read by customer processes. Would be curious to hear the trade-offs involved in that decision, they could not have made it lightly, I always try to choose fail-safe design modes and at that level of solutioning I'm sure their teams are way smarter than I am so I'd love to learn from this use case.
Also interesting to find out AWS uses their customers' resources for temporary storage in the transit process, instead of elastically using process-only-bound ephemeral resources outside customer space in a more cloud-native fashion. Temporary consumption in the customers' resource space in a solution pattern gives me nightmare scenarios of stray code that scribbles the temporary objects into customer-owned data, or accidentally dropped into the wrong location and read by customer processes. Would be curious to hear the trade-offs involved in that decision, they could not have made it lightly, I always try to choose fail-safe design modes and at that level of solutioning I'm sure their teams are way smarter than I am so I'd love to learn from this use case.