Ha, thanks for the feedback! People have made a few points about the styling, it definitely needs a harder look. Maybe a silly question but which do you find worse, the blue color or the underlines?
Companies charge what the market can bear, not based on their costs. Certainly they will often use some multiplier of their costs as a starting point, and they can't sustainably charge below their costs. But if they double the price of the product and lose more than half of their customers, that's a failure to set pricing properly.
Consider the reverse direction: if a company can decrease costs, they will usually pocket the extra profit, not reduce the price they charge. Price cuts usually only happen for one of two reasons: 1) to avoid losing customers to another company that is charging less (or to entice customers of another company that's charging the same), or 2) to capture more profit if they'll earn more customers at a lower price than they'll lose due to the lower per-unit profit. (Yes, there are other reasons, but these seem to be the main ones.
For goods that are not essential to life, prices are set based on what people are willing to pay vs. how many units can be sold at that price, with the cost as a floor (absent a policy of using a product as a loss leader).
Server will store encrypted blob and its hash/etag.
Client before upload of data, check for hash/etag of blob he originally fetched. If blob on server has different one, it will download it, decrypt, patch new data on existing one, encrypt and reupload.
Whats the catch?
AES is hardware accelerated on the most devices - so with all the ops it will be significantly faster than any homomorphic enc nowadays.
I too was wondering the same thing. FHE is cool tech, but this seems to me to be a bad application of it since it will undoubtedly be less efficient.
FHE is useful when trying to compute on data from various sources who all mutually want to keep some information secret. For example, Apple's use of FHE to categorize photos [1]. In this case all the server is really doing is "compressing" for lack of a better word, the change sets, so each offline client doesn't need to sync every message since they are already merged by the server.
If all you want is to keep a synchronizing server in the dark, but all clients can be trusted with the unencrypted data, traditional key exchange and symmetric encryption should suffice.
For now, unfortunately, no - no signed URLs are supported. It wasn't my focus (use case), but if you find a simple/minimalistic way to implement it, I can help you with that to integrate it.
From my helicopter perspective, it adds extra complexity and size, which could maybe be ideal for a separate fork/project?
Signed URLs are great because it allows you to allow third parties access to a file without them having to authenticate against AWS.
Our primary use case is browser-based uploads. You don't want people uploading anything and everything, like the wordpress upload folder. And it's timed, so you don't have to worry about someone recycling the URL.
I use presigned urls as part of a federation layer on top of an s3 bucket. Users make authenticated requests to my api which checks their permissions (if they have access to read/write to the specified slice of the s3 bucket), my api sends a presigned url back to allow read/write/delete to that specific portion of the bucket.
Good to see this mentioned. We are considering running it for some things internally, along with Harbor. The fact that the resource footprint is advertised as small enough is compelling.
checksumming does make sense because it ensures that the file you've transferred is complete and what was expected. if the checksum of the file you've downloaded differs from the server gave you, you should not process the file further and throw an error (worst case would probably be a man in the middle attack, not so worse cases being packet loss i guess)
> checksumming does make sense because it ensures that the file you've transferred is complete and what was expected.
TCP has a checksum for packet loss, and TLS protects against MITM.
I've always found this aspect of S3's design questionable. Sending both a content-md5 AND a x-amz-content-sha256 header and taking up gobs of compute in the process, sheesh...
It's also part of the reason why running minio in its single node single drive mode is a resource hog.
Effingo file copy service does application-layer strong checksums and detects about 4.5 corruptions per exabyte transferred (figure 9, section 6.2 in [1]).
This is on top of TCP checksums, transport layer checksums/encryption (gRPC), ECC RAM and other layers along the way.
Many of these could be traced back to a "broken" machine that was eventually taken out.
In my view one reason is to ensure integrity down the line. You want the checksum of a file to still be the same when you download it maybe years later. If it isn't, you get warned about it. Without the checksum, how will you know for sure? Keep your own database of checksums? :)
If we're talking about bitrot protection, I'm pretty sure S3 would use some form of checksum (such as crc32 or xxhash) on each internal block to facilitate the Reed-Solomon process.
If it's verifying whether if it's the same file, you can use the Etag header which is computed server side by S3. Although I don't like this design as it ossifies the checksum algorithm.
This is actually not the case. The TLS stream ensures that the packets transferred between your machine and S3 are not corrupted, but that doesn't protect against bit-flips which could (though, obviously, shouldn't) occur from within S3 itself. The benefit of an end-to-end checksum like this is that the S3 system can store it directly next to the data, validate it when it reads the data back (making sure that nothing has changed since your original PutObject), and then give it back to you on request (so that you can also validate it in your client). It's the only way for your client to have bullet-proof certainty of integrity the entire time that the data is in the system.
Thats true, but wouldn't it be still required if you're having a internal S3 service which is used by internal services and does not have HTTPS (as it is not exposed to the public)? I get that the best practice would be to also use HTTPS there but I'd guess thats not the norm?
Theoretically TCP packets have checksums, however it's fairly weak. So for HTTP, additional checksums make sense. Although I'm not sure, if there are any internal AWS S3 deployments working over HTTP and why would they complicate their protocol for everyone to help such a niche use case.
I'm sure that they have reasons for this whole request signature scheme over traditional "Authorization: Bearer $token" header, but I never understood it.
AWS has a video about it somewhere, but in general, it’s because S3 was designed in a world where not all browsers/clients had HTTPS and it was a reasonably expensive operation to do the encryption (like, IE6 world). SigV4 (and its predecessors) are cheap and easy once you understand the code.
Because a bearer token is a bearer token to do any request, while a pre-signed request allows you to hand out the capability to perform _only that specific request_.
On the other hand S3 uses checksums only to verify expected upload (on the write from client -> server) ... and suprisingly you can do that in paralel after the upload - by checking the MD5 hash of blob to ETag (*with some caveats)
You need the checksum only if the file is big and you're downloading it to disk, or if you're paranoid that some malware with root access might be altering the contents of your memory.
Or you really care about the data and are aware of the statistical inevitability of a bit flip somewhere along the line if you’re operating long enough.
reply