> These tests were run against a copy of the live database in a production environment.
This is pretty terrible from a security standpoint. A development environment is typically much less secure than the live environment, and for good reason. The development environment must be accessible to developers, typically both on-site and remote. All developers have access to test databases for the purpose of testing their changes. There are often many more software packages in a development environment, and development servers have a higher probability of running vulnerable services. Live environments typically have much better logging and auditing.
Every company should have a program that can be run to sanitize the live database for use in testing. I've seen too many situations where the production environment was appropriately locked down and audited, but the development environment was compromised. It's not unheard-of for a developer to lose possession of his laptop, and if it contains a copy of the live database it's no better than the site, itself, being compromised.
You raise a great point that I wished I had clarified. I worked on the product side of the store, not, say order fulfillment, order integration, etc.
Multiple databases made up the live environment. The developers did not have access to the live environment. As a matter of fact, we had developed it to the point that, except in special circumstances, the engineers didn't even know what products were about to be announced until Steve Jobs announced them to the public.
The dev database was a copy of the products database – most everything in the products database was either public information or it was obsolete data for EOL Apple and third party products. I don't recall ever seeing any sensitive data in our production environment. Unlike, say Walmart or Amazon, we didn't need to keep track of inventory since we build the products (the only exception that comes to mind were the refurbed products since there was a limited amount of inventory).
Dev systems would process dummy credit card numbers, etc, in dev (but I didn't work, in that area). The applications were smart enough to figure out which environment they were in when started up which is a big help.
Every company should have a program that can be run to sanitize the live database for use in testing.
+1.
However, modifying or sometimes even a dump/load on a SQL database can change its behavior dramatically.
But the harder part can be getting a hardware setup similar to production. Most places want their newest, best hardware in production for obvious reasons. Few places want to pay for mirror-image production systems just for testing to bang on.
There's the old saying: If you have two systems, 'production' and 'testing', the one with the more challenging load is 'testing'.
I have to second BrandomM's comments. Don't make a complete copy of your live database to use in dev. Sensitive data needs to be scrubbed, ideally by a bona fide DBA, before bringing it over. Best to avoid this if at all possible so you don't have to worry about overlooking something that was sensitive.
Also, I don't recall ever seeing any security issues with our environment or how we handled the code and data. There were some very smart engineers "minding the store."
I agree. My main intention was to inform would-be copy-cats that they shouldn't play fast and loose with copies of their live database. I'm sure (or at least, I hope) that Apple is smarter about sensitive data than that.
> These tests were run against a copy of the live database in a production environment.
Read above line twice. He is saying that test is done in 'production environment'. Do not assume there is 'development environment' for testing within 'production environment'
The "production environment" is obviously not a "live production environment"; it's some kind of development environment.
And mostly, I made my comment in the hopes that people here wouldn't go, "Oh, Apple tests against a copy of the live database. I should do that, too!" I'm sure Apple is smarter about it than my comment indicated.
My point was simply to be careful with the live database. Investing in state-of-the-art protection and auditing in your live environment can be trivially defeated by simply copying the live database to an insecure environment. So don't do it.
In my experience, large high tech organizations have many environments such as dev, build, QA, staging, demo, live, etc. The really good organization, with the smart engineers, have built this environments over the span of years and they've carefully thought through how each environment works.
Also, some organizations, especially ones that use cloud computing (EC2, Slicehost, etc) have the option of creating a staging environment for QA. testing, etc and, if everything checks out, the staging environment can be restarted as a live environment (there are ways to do this safely and securely) while leaving the previous live environment live. If, for some reason, you find problems with the new live environment, then you can simply shut it down and keep the old live environment humming along. Of course, this is easier said than done, but it'll keep the anxiety level down.
Another good idea is to have apps that know which environment they're in when they start up. So, if an app starts up in dev, it should only access the dev database. If it starts up in the live deployment environment, it should only access the live database. To enforce this security, the live database must only allow connections from clients in the live environment (IP address).
But, these "self aware" apps don't necessarily need (probably should not) have configuration files with the sensitive data used to access databases, credit card processors, etc. Rather, at app start up, these apps should know which environment they're in and then go over the network to fetch the sensitive data used for logging into databases, services, etc. (The apps should only store this sensitive data in memory.) The services, on the other end of that connection, should be checking the IP address of the app which is asking for the data to ensure that there isn't a mixup.
This is pretty terrible from a security standpoint. A development environment is typically much less secure than the live environment, and for good reason. The development environment must be accessible to developers, typically both on-site and remote. All developers have access to test databases for the purpose of testing their changes. There are often many more software packages in a development environment, and development servers have a higher probability of running vulnerable services. Live environments typically have much better logging and auditing.
Every company should have a program that can be run to sanitize the live database for use in testing. I've seen too many situations where the production environment was appropriately locked down and audited, but the development environment was compromised. It's not unheard-of for a developer to lose possession of his laptop, and if it contains a copy of the live database it's no better than the site, itself, being compromised.