s3实现视频分片预览

Posted on Edited on In Views:
在一些支持分段获取文件的服务中，通过Range获取指定分片，可以分段获取文件而不用将文件整个下载下来。

s3 对象存储

微服务架构下，通常会将s3对象存储作为后端存储，而更多的是使用云厂商的对象存储，例如亚马逊s3、腾讯COS、七牛云OSS，这些云存储都有完善的SDK。还有一些会使用开源软件提供的对象存储协议和接口，例如Ceph支持对象存储，Minio支持将带有文件系统的目录映射成对象存储，也能对外提供对象存储协议和接口。

AWS SDK for Go

https://github.com/aws/aws-sdk-go

示例上传文件代码：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
 package main

 import (
     "context"
     "flag"
     "fmt"
     "os"
     "time"

     "github.com/aws/aws-sdk-go/aws"
     "github.com/aws/aws-sdk-go/aws/awserr"
     "github.com/aws/aws-sdk-go/aws/request"
     "github.com/aws/aws-sdk-go/aws/session"
     "github.com/aws/aws-sdk-go/service/s3"
 )

 // Uploads a file to S3 given a bucket and object key. Also takes a duration
 // value to terminate the update if it doesn't complete within that time.
 //
 // The AWS Region needs to be provided in the AWS shared config or on the
 // environment variable as `AWS_REGION`. Credentials also must be provided
 // Will default to shared config file, but can load from environment if provided.
 //
 // Usage:
 //   # Upload myfile.txt to myBucket/myKey. Must complete within 10 minutes or will fail
 //   go run withContext.go -b mybucket -k myKey -d 10m < myfile.txt
 func main() {
     var bucket, key string
     var timeout time.Duration

     flag.StringVar(&bucket, "b", "", "Bucket name.")
     flag.StringVar(&key, "k", "", "Object key name.")
     flag.DurationVar(&timeout, "d", 0, "Upload timeout.")
     flag.Parse()

     // All clients require a Session. The Session provides the client with
    // shared configuration such as region, endpoint, and credentials. A
    // Session should be shared where possible to take advantage of
    // configuration and credential caching. See the session package for
    // more information.
     sess := session.Must(session.NewSession())

    // Create a new instance of the service's client with a Session.
    // Optional aws.Config values can also be provided as variadic arguments
    // to the New function. This option allows you to provide service
    // specific configuration.
     svc := s3.New(sess)

     // Create a context with a timeout that will abort the upload if it takes
     // more than the passed in timeout.
     ctx := context.Background()
     var cancelFn func()
     if timeout > 0 {
         ctx, cancelFn = context.WithTimeout(ctx, timeout)
     }
     // Ensure the context is canceled to prevent leaking.
     // See context package for more information, https://golang.org/pkg/context/
if cancelFn != nil {
         defer cancelFn()
}

     // Uploads the object to S3. The Context will interrupt the request if the
     // timeout expires.
     _, err := svc.PutObjectWithContext(ctx, &s3.PutObjectInput{
         Bucket: aws.String(bucket),
         Key:    aws.String(key),
         Body:   os.Stdin,
     })
     if err != nil {
         if aerr, ok := err.(awserr.Error); ok && aerr.Code() == request.CanceledErrorCode {
             // If the SDK can determine the request or retry delay was canceled
             // by a context the CanceledErrorCode error code will be returned.
             fmt.Fprintf(os.Stderr, "upload canceled due to timeout, %v\n", err)
         } else {
             fmt.Fprintf(os.Stderr, "failed to upload object, %v\n", err)
         }
         os.Exit(1)
     }

     fmt.Printf("successfully uploaded file to %s/%s\n", bucket, key)
 }

现在出了v2版本，更推荐使用高版本

AWS SDK for Go v2

https://github.com/aws/aws-sdk-go-v2

https://aws.github.io/aws-sdk-go-v2/docs/

示例代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
)

func main() {
    // Using the SDK's default configuration, loading additional config
    // and credentials values from the environment variables, shared
    // credentials, and shared configuration files
    cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion("us-west-2"))
    if err != nil {
        log.Fatalf("unable to load SDK config, %v", err)
    }

    // Using the Config value, create the DynamoDB client
    svc := dynamodb.NewFromConfig(cfg)

    // Build the request with its input parameters
    resp, err := svc.ListTables(context.TODO(), &dynamodb.ListTablesInput{
        Limit: aws.Int32(5),
    })
    if err != nil {
        log.Fatalf("failed to list tables, %v", err)
    }

    fmt.Println("Tables:")
    for _, tableName := range resp.TableNames {
        fmt.Println(tableName)
    }
}

使用SDK的两种方式

直接通过SDK实现上传和下载

直接通过SDK实现上传和下载的方式在aws-sdk-go和aws-sdk-go-v2中都可实现，入参和出参不同。

上传文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
// Adds an object to a bucket. You must have WRITE permissions on a bucket to add
// an object to it. Amazon S3 never adds partial objects; if you receive a success
// response, Amazon S3 added the entire object to the bucket. You cannot use
// PutObject to only update a single piece of metadata for an existing object. You
// must put the entire object with updated metadata if you want to update some
// values. Amazon S3 is a distributed system. If it receives multiple write
// requests for the same object simultaneously, it overwrites all but the last
// object written. To prevent objects from being deleted or overwritten, you can
// use Amazon S3 Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html)
// . To ensure that data is not corrupted traversing the network, use the
// Content-MD5 header. When you use this header, Amazon S3 checks the object
// against the provided MD5 value and, if they do not match, returns an error.
// Additionally, you can calculate the MD5 while putting an object to Amazon S3 and
// compare the returned ETag to the calculated MD5 value.
//   - To successfully complete the PutObject request, you must have the
//     s3:PutObject in your IAM permissions.
//   - To successfully change the objects acl of your PutObject request, you must
//     have the s3:PutObjectAcl in your IAM permissions.
//   - To successfully set the tag-set with your PutObject request, you must have
//     the s3:PutObjectTagging in your IAM permissions.
//   - The Content-MD5 header is required for any request to upload an object with
//     a retention period configured using Amazon S3 Object Lock. For more information
//     about Amazon S3 Object Lock, see Amazon S3 Object Lock Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html)
//     in the Amazon S3 User Guide.
//
// You have three mutually exclusive options to protect data using server-side
// encryption in Amazon S3, depending on how you choose to manage the encryption
// keys. Specifically, the encryption key options are Amazon S3 managed keys
// (SSE-S3), Amazon Web Services KMS keys (SSE-KMS), and customer-provided keys
// (SSE-C). Amazon S3 encrypts data with server-side encryption by using Amazon S3
// managed keys (SSE-S3) by default. You can optionally tell Amazon S3 to encrypt
// data at by rest using server-side encryption with other key options. For more
// information, see Using Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html)
// . When adding a new object, you can use headers to grant ACL-based permissions
// to individual Amazon Web Services accounts or to predefined groups defined by
// Amazon S3. These permissions are then added to the ACL on the object. By
// default, all objects are private. Only the owner has full access control. For
// more information, see Access Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html)
// and Managing ACLs Using the REST API (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-using-rest-api.html)
// . If the bucket that you're uploading objects to uses the bucket owner enforced
// setting for S3 Object Ownership, ACLs are disabled and no longer affect
// permissions. Buckets that use this setting only accept PUT requests that don't
// specify an ACL or PUT requests that specify bucket owner full control ACLs, such
// as the bucket-owner-full-control canned ACL or an equivalent form of this ACL
// expressed in the XML format. PUT requests that contain other ACLs (for example,
// custom grants to certain Amazon Web Services accounts) fail and return a 400
// error with the error code AccessControlListNotSupported . For more information,
// see Controlling ownership of objects and disabling ACLs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/about-object-ownership.html)
// in the Amazon S3 User Guide. If your bucket uses the bucket owner enforced
// setting for Object Ownership, all objects written to the bucket by any account
// will be owned by the bucket owner. By default, Amazon S3 uses the STANDARD
// Storage Class to store newly created objects. The STANDARD storage class
// provides high durability and high availability. Depending on performance needs,
// you can specify a different Storage Class. Amazon S3 on Outposts only uses the
// OUTPOSTS Storage Class. For more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html)
// in the Amazon S3 User Guide. If you enable versioning for a bucket, Amazon S3
// automatically generates a unique version ID for the object being stored. Amazon
// S3 returns this ID in the response. When you enable versioning for a bucket, if
// Amazon S3 receives multiple write requests for the same object simultaneously,
// it stores all of the objects. For more information about versioning, see Adding
// Objects to Versioning-Enabled Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/AddingObjectstoVersioningEnabledBuckets.html)
// . For information about returning the versioning state of a bucket, see
// GetBucketVersioning (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketVersioning.html)
// . For more information about related Amazon S3 APIs, see the following:
//   - CopyObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html)
//   - DeleteObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html)
func (c *Client) PutObject(ctx context.Context, params *PutObjectInput, optFns ...func(*Options)) (*PutObjectOutput, error) {
    if params == nil {
        params = &PutObjectInput{}
    }

    result, metadata, err := c.invokeOperation(ctx, "PutObject", params, optFns, c.addOperationPutObjectMiddlewares)
    if err != nil {
        return nil, err
    }

    out := result.(*PutObjectOutput)
    out.ResultMetadata = metadata
    return out, nil
}

入参

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
type PutObjectInput struct {

    // The bucket name to which the PUT action was initiated. When using this action
    // with an access point, you must direct requests to the access point hostname. The
    // access point hostname takes the form
    // AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. When using this
    // action with an access point through the Amazon Web Services SDKs, you provide
    // the access point ARN in place of the bucket name. For more information about
    // access point ARNs, see Using access points (https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-access-points.html)
    // in the Amazon S3 User Guide. When you use this action with Amazon S3 on
    // Outposts, you must direct requests to the S3 on Outposts hostname. The S3 on
    // Outposts hostname takes the form
    // AccessPointName-AccountId.outpostID.s3-outposts.Region.amazonaws.com . When you
    // use this action with S3 on Outposts through the Amazon Web Services SDKs, you
    // provide the Outposts access point ARN in place of the bucket name. For more
    // information about S3 on Outposts ARNs, see What is S3 on Outposts (https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3onOutposts.html)
    // in the Amazon S3 User Guide.
    //
    // This member is required.
    Bucket *string

    // Object key for which the PUT action was initiated.
    //
    // This member is required.
    Key *string

    // The canned ACL to apply to the object. For more information, see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL)
    // . This action is not supported by Amazon S3 on Outposts.
    ACL types.ObjectCannedACL

    // Object data.
    Body io.Reader

    // Specifies whether Amazon S3 should use an S3 Bucket Key for object encryption
    // with server-side encryption using AWS KMS (SSE-KMS). Setting this header to true
    // causes Amazon S3 to use an S3 Bucket Key for object encryption with SSE-KMS.
    // Specifying this header with a PUT action doesn’t affect bucket-level settings
    // for S3 Bucket Key.
    BucketKeyEnabled bool

    // Can be used to specify caching behavior along the request/reply chain. For more
    // information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9)
    // .
    CacheControl *string

    // Indicates the algorithm used to create the checksum for the object when using
    // the SDK. This header will not provide any additional functionality if not using
    // the SDK. When sending this header, there must be a corresponding x-amz-checksum
    // or x-amz-trailer header sent. Otherwise, Amazon S3 fails the request with the
    // HTTP status code 400 Bad Request . For more information, see Checking object
    // integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide. If you provide an individual checksum, Amazon S3
    // ignores any provided ChecksumAlgorithm parameter.
    ChecksumAlgorithm types.ChecksumAlgorithm

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 32-bit CRC32 checksum of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumCRC32 *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 32-bit CRC32C checksum of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumCRC32C *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 160-bit SHA-1 digest of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumSHA1 *string

    // This header can be used as a data integrity check to verify that the data
    // received is the same data that was originally sent. This header specifies the
    // base64-encoded, 256-bit SHA-256 digest of the object. For more information, see
    // Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html)
    // in the Amazon S3 User Guide.
    ChecksumSHA256 *string

    // Specifies presentational information for the object. For more information, see
    // https://www.rfc-editor.org/rfc/rfc6266#section-4 (https://www.rfc-editor.org/rfc/rfc6266#section-4)
    // .
    ContentDisposition *string

    // Specifies what content encodings have been applied to the object and thus what
    // decoding mechanisms must be applied to obtain the media-type referenced by the
    // Content-Type header field. For more information, see
    // https://www.rfc-editor.org/rfc/rfc9110.html#field.content-encoding (https://www.rfc-editor.org/rfc/rfc9110.html#field.content-encoding)
    // .
    ContentEncoding *string

    // The language the content is in.
    ContentLanguage *string

    // Size of the body in bytes. This parameter is useful when the size of the body
    // cannot be determined automatically. For more information, see
    // https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length (https://www.rfc-editor.org/rfc/rfc9110.html#name-content-length)
    // .
    ContentLength int64

    // The base64-encoded 128-bit MD5 digest of the message (without the headers)
    // according to RFC 1864. This header can be used as a message integrity check to
    // verify that the data is the same data that was originally sent. Although it is
    // optional, we recommend using the Content-MD5 mechanism as an end-to-end
    // integrity check. For more information about REST request authentication, see
    // REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html)
    // .
    ContentMD5 *string

    // A standard MIME type describing the format of the contents. For more
    // information, see https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type (https://www.rfc-editor.org/rfc/rfc9110.html#name-content-type)
    // .
    ContentType *string

    // The account ID of the expected bucket owner. If the bucket is owned by a
    // different account, the request fails with the HTTP status code 403 Forbidden
    // (access denied).
    ExpectedBucketOwner *string

    // The date and time at which the object is no longer cacheable. For more
    // information, see https://www.rfc-editor.org/rfc/rfc7234#section-5.3 (https://www.rfc-editor.org/rfc/rfc7234#section-5.3)
    // .
    Expires *time.Time

    // Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object. This
    // action is not supported by Amazon S3 on Outposts.
    GrantFullControl *string

    // Allows grantee to read the object data and its metadata. This action is not
    // supported by Amazon S3 on Outposts.
    GrantRead *string

    // Allows grantee to read the object ACL. This action is not supported by Amazon
    // S3 on Outposts.
    GrantReadACP *string

    // Allows grantee to write the ACL for the applicable object. This action is not
    // supported by Amazon S3 on Outposts.
    GrantWriteACP *string

    // A map of metadata to store with the object in S3.
    Metadata map[string]string

    // Specifies whether a legal hold will be applied to this object. For more
    // information about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html)
    // .
    ObjectLockLegalHoldStatus types.ObjectLockLegalHoldStatus

    // The Object Lock mode that you want to apply to this object.
    ObjectLockMode types.ObjectLockMode

    // The date and time when you want this object's Object Lock to expire. Must be
    // formatted as a timestamp parameter.
    ObjectLockRetainUntilDate *time.Time

    // Confirms that the requester knows that they will be charged for the request.
    // Bucket owners need not specify this parameter in their requests. For information
    // about downloading objects from Requester Pays buckets, see Downloading Objects
    // in Requester Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html)
    // in the Amazon S3 User Guide.
    RequestPayer types.RequestPayer

    // Specifies the algorithm to use to when encrypting the object (for example,
    // AES256).
    SSECustomerAlgorithm *string

    // Specifies the customer-provided encryption key for Amazon S3 to use in
    // encrypting data. This value is used to store the object and then it is
    // discarded; Amazon S3 does not store the encryption key. The key must be
    // appropriate for use with the algorithm specified in the
    // x-amz-server-side-encryption-customer-algorithm header.
    SSECustomerKey *string

    // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321.
    // Amazon S3 uses this header for a message integrity check to ensure that the
    // encryption key was transmitted without error.
    SSECustomerKeyMD5 *string

    // Specifies the Amazon Web Services KMS Encryption Context to use for object
    // encryption. The value of this header is a base64-encoded UTF-8 string holding
    // JSON with the encryption context key-value pairs. This value is stored as object
    // metadata and automatically gets passed on to Amazon Web Services KMS for future
    // GetObject or CopyObject operations on this object.
    SSEKMSEncryptionContext *string

    // If x-amz-server-side-encryption has a valid value of aws:kms , this header
    // specifies the ID of the Amazon Web Services Key Management Service (Amazon Web
    // Services KMS) symmetric encryption customer managed key that was used for the
    // object. If you specify x-amz-server-side-encryption:aws:kms , but do not provide
    // x-amz-server-side-encryption-aws-kms-key-id , Amazon S3 uses the Amazon Web
    // Services managed key to protect the data. If the KMS key does not exist in the
    // same account issuing the command, you must use the full ARN and not just the ID.
    SSEKMSKeyId *string

    // The server-side encryption algorithm used when storing this object in Amazon S3
    // (for example, AES256, aws:kms ).
    ServerSideEncryption types.ServerSideEncryption

    // By default, Amazon S3 uses the STANDARD Storage Class to store newly created
    // objects. The STANDARD storage class provides high durability and high
    // availability. Depending on performance needs, you can specify a different
    // Storage Class. Amazon S3 on Outposts only uses the OUTPOSTS Storage Class. For
    // more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html)
    // in the Amazon S3 User Guide.
    StorageClass types.StorageClass

    // The tag-set for the object. The tag-set must be encoded as URL Query
    // parameters. (For example, "Key1=Value1")
    Tagging *string

    // If the bucket is configured as a website, redirects requests for this object to
    // another object in the same bucket or to an external URL. Amazon S3 stores the
    // value of this header in the object metadata. For information about object
    // metadata, see Object Key and Metadata (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html)
    // . In the following example, the request header sets the redirect to an object
    // (anotherPage.html) in the same bucket: x-amz-website-redirect-location:
    // /anotherPage.html In the following example, the request header sets the object
    // redirect to another website: x-amz-website-redirect-location:
    // http://www.example.com/ For more information about website hosting in Amazon S3,
    // see Hosting Websites on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html)
    // and How to Configure Website Page Redirects (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html)
    // .
    WebsiteRedirectLocation *string

    noSmithyDocumentSerde
}

直接通过函数上传的方式可以看到，文件的一些元数据信息和缓存信息都在入参中，文件的内容以Body io.Reader的形式放在入参中。

下载

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
// Retrieves objects from Amazon S3. To use GET , you must have READ access to the
// object. If you grant READ access to the anonymous user, you can return the
// object without using an authorization header. An Amazon S3 bucket has no
// directory hierarchy such as you would find in a typical computer file system.
// You can, however, create a logical hierarchy by using object key names that
// imply a folder structure. For example, instead of naming an object sample.jpg ,
// you can name it photos/2006/February/sample.jpg . To get an object from such a
// logical hierarchy, specify the full key name for the object in the GET
// operation. For a virtual hosted-style request example, if you have the object
// photos/2006/February/sample.jpg , specify the resource as
// /photos/2006/February/sample.jpg . For a path-style request example, if you have
// the object photos/2006/February/sample.jpg in the bucket named examplebucket ,
// specify the resource as /examplebucket/photos/2006/February/sample.jpg . For
// more information about request types, see HTTP Host Header Bucket Specification (https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket)
// . For more information about returning the ACL of an object, see GetObjectAcl (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html)
// . If the object you are retrieving is stored in the S3 Glacier or S3 Glacier
// Deep Archive storage class, or S3 Intelligent-Tiering Archive or S3
// Intelligent-Tiering Deep Archive tiers, before you can retrieve the object you
// must first restore a copy using RestoreObject (https://docs.aws.amazon.com/AmazonS3/latest/API/API_RestoreObject.html)
// . Otherwise, this action returns an InvalidObjectState error. For information
// about restoring archived objects, see Restoring Archived Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html)
// . Encryption request headers, like x-amz-server-side-encryption , should not be
// sent for GET requests if your object uses server-side encryption with KMS keys
// (SSE-KMS) or server-side encryption with Amazon S3–managed encryption keys
// (SSE-S3). If your object does use these types of keys, you’ll get an HTTP 400
// BadRequest error. If you encrypt an object by using server-side encryption with
// customer-provided encryption keys (SSE-C) when you store the object in Amazon
// S3, then when you GET the object, you must use the following headers:
//   - x-amz-server-side-encryption-customer-algorithm
//   - x-amz-server-side-encryption-customer-key
//   - x-amz-server-side-encryption-customer-key-MD5
//
// For more information about SSE-C, see Server-Side Encryption (Using
// Customer-Provided Encryption Keys) (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html)
// . Assuming you have the relevant permission to read object tags, the response
// also returns the x-amz-tagging-count header that provides the count of number
// of tags associated with the object. You can use GetObjectTagging (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html)
// to retrieve the tag set associated with an object. Permissions You need the
// relevant read object (or version) permission for this operation. For more
// information, see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html)
// . If the object you request does not exist, the error Amazon S3 returns depends
// on whether you also have the s3:ListBucket permission.
//   - If you have the s3:ListBucket permission on the bucket, Amazon S3 will
//     return an HTTP status code 404 ("no such key") error.
//   - If you don’t have the s3:ListBucket permission, Amazon S3 will return an
//     HTTP status code 403 ("access denied") error.
//
// Versioning By default, the GET action returns the current version of an object.
// To return a different version, use the versionId subresource.
//   - If you supply a versionId , you need the s3:GetObjectVersion permission to
//     access a specific version of an object. If you request a specific version, you
//     do not need to have the s3:GetObject permission. If you request the current
//     version without a specific version ID, only s3:GetObject permission is
//     required. s3:GetObjectVersion permission won't be required.
//   - If the current version of the object is a delete marker, Amazon S3 behaves
//     as if the object was deleted and includes x-amz-delete-marker: true in the
//     response.
//
// For more information about versioning, see PutBucketVersioning (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketVersioning.html)
// . Overriding Response Header Values There are times when you want to override
// certain response header values in a GET response. For example, you might
// override the Content-Disposition response header value in your GET request. You
// can override values for a set of response headers using the following query
// parameters. These response header values are sent only on a successful request,
// that is, when status code 200 OK is returned. The set of headers you can
// override using these parameters is a subset of the headers that Amazon S3
// accepts when you create an object. The response headers that you can override
// for the GET response are Content-Type , Content-Language , Expires ,
// Cache-Control , Content-Disposition , and Content-Encoding . To override these
// header values in the GET response, you use the following request parameters. You
// must sign the request, either using an Authorization header or a presigned URL,
// when using these parameters. They cannot be used with an unsigned (anonymous)
// request.
//   - response-content-type
//   - response-content-language
//   - response-expires
//   - response-cache-control
//   - response-content-disposition
//   - response-content-encoding
//
// Overriding Response Header Values If both of the If-Match and
// If-Unmodified-Since headers are present in the request as follows: If-Match
// condition evaluates to true , and; If-Unmodified-Since condition evaluates to
// false ; then, S3 returns 200 OK and the data requested. If both of the
// If-None-Match and If-Modified-Since headers are present in the request as
// follows: If-None-Match condition evaluates to false , and; If-Modified-Since
// condition evaluates to true ; then, S3 returns 304 Not Modified response code.
// For more information about conditional requests, see RFC 7232 (https://tools.ietf.org/html/rfc7232)
// . The following operations are related to GetObject :
//   - ListBuckets (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html)
//   - GetObjectAcl (https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAcl.html)
func (c *Client) GetObject(ctx context.Context, params *GetObjectInput, optFns ...func(*Options)) (*GetObjectOutput, error) {
    if params == nil {
        params = &GetObjectInput{}
    }

    result, metadata, err := c.invokeOperation(ctx, "GetObject", params, optFns, c.addOperationGetObjectMiddlewares)
    if err != nil {
        return nil, err
    }

    out := result.(*GetObjectOutput)
    out.ResultMetadata = metadata
    return out, nil
}

出参

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
type GetObjectOutput struct {

    // Indicates that a range of bytes was specified.
    AcceptRanges *string

    // Object data.
    Body io.ReadCloser

    // Indicates whether the object uses an S3 Bucket Key for server-side encryption
    // with Amazon Web Services KMS (SSE-KMS).
    BucketKeyEnabled bool

    // Specifies caching behavior along the request/reply chain.
    CacheControl *string

    // The base64-encoded, 32-bit CRC32 checksum of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32 *string

    // The base64-encoded, 32-bit CRC32C checksum of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumCRC32C *string

    // The base64-encoded, 160-bit SHA-1 digest of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA1 *string

    // The base64-encoded, 256-bit SHA-256 digest of the object. This will only be
    // present if it was uploaded with the object. With multipart uploads, this may not
    // be a checksum value of the object. For more information about how checksums are
    // calculated with multipart uploads, see Checking object integrity (https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums)
    // in the Amazon S3 User Guide.
    ChecksumSHA256 *string

    // Specifies presentational information for the object.
    ContentDisposition *string

    // Specifies what content encodings have been applied to the object and thus what
    // decoding mechanisms must be applied to obtain the media-type referenced by the
    // Content-Type header field.
    ContentEncoding *string

    // The language the content is in.
    ContentLanguage *string

    // Size of the body in bytes.
    ContentLength int64

    // The portion of the object returned in the response.
    ContentRange *string

    // A standard MIME type describing the format of the object data.
    ContentType *string

    // Specifies whether the object retrieved was (true) or was not (false) a Delete
    // Marker. If false, this response header does not appear in the response.
    DeleteMarker bool

    // An entity tag (ETag) is an opaque identifier assigned by a web server to a
    // specific version of a resource found at a URL.
    ETag *string

    // If the object expiration is configured (see PUT Bucket lifecycle), the response
    // includes this header. It includes the expiry-date and rule-id key-value pairs
    // providing object expiration information. The value of the rule-id is
    // URL-encoded.
    Expiration *string

    // The date and time at which the object is no longer cacheable.
    Expires *time.Time

    // Creation date of the object.
    LastModified *time.Time

    // A map of metadata to store with the object in S3.
    //
    // Map keys will be normalized to lower-case.
    Metadata map[string]string

    // This is set to the number of metadata entries not returned in x-amz-meta
    // headers. This can happen if you create metadata using an API like SOAP that
    // supports more flexible metadata than the REST API. For example, using SOAP, you
    // can create metadata whose values are not legal HTTP headers.
    MissingMeta int32

    // Indicates whether this object has an active legal hold. This field is only
    // returned if you have permission to view an object's legal hold status.
    ObjectLockLegalHoldStatus types.ObjectLockLegalHoldStatus

    // The Object Lock mode currently in place for this object.
    ObjectLockMode types.ObjectLockMode

    // The date and time when this object's Object Lock will expire.
    ObjectLockRetainUntilDate *time.Time

    // The count of parts this object has. This value is only returned if you specify
    // partNumber in your request and the object was uploaded as a multipart upload.
    PartsCount int32

    // Amazon S3 can return this if your request involves a bucket that is either a
    // source or destination in a replication rule.
    ReplicationStatus types.ReplicationStatus

    // If present, indicates that the requester was successfully charged for the
    // request.
    RequestCharged types.RequestCharged

    // Provides information about object restoration action and expiration time of the
    // restored object copy.
    Restore *string

    // If server-side encryption with a customer-provided encryption key was
    // requested, the response will include this header confirming the encryption
    // algorithm used.
    SSECustomerAlgorithm *string

    // If server-side encryption with a customer-provided encryption key was
    // requested, the response will include this header to provide round-trip message
    // integrity verification of the customer-provided encryption key.
    SSECustomerKeyMD5 *string

    // If present, specifies the ID of the Amazon Web Services Key Management Service
    // (Amazon Web Services KMS) symmetric encryption customer managed key that was
    // used for the object.
    SSEKMSKeyId *string

    // The server-side encryption algorithm used when storing this object in Amazon S3
    // (for example, AES256, aws:kms ).
    ServerSideEncryption types.ServerSideEncryption

    // Provides storage class information of the object. Amazon S3 returns this header
    // for all objects except for S3 Standard storage class objects.
    StorageClass types.StorageClass

    // The number of tags, if any, on the object.
    TagCount int32

    // Version of the object.
    VersionId *string

    // If the bucket is configured as a website, redirects requests for this object to
    // another object in the same bucket or to an external URL. Amazon S3 stores the
    // value of this header in the object metadata.
    WebsiteRedirectLocation *string

    // Metadata pertaining to the operation's result.
    ResultMetadata middleware.Metadata

    noSmithyDocumentSerde
}

直接通过函数下载的方式可以看到，文件的一些元数据信息和缓存信息都在出参中，文件的内容以Body io.ReadCloser的形式放在入参中。

通过HTTP请求实现上传和下载

通过HTTP请求实现上传和下载只有在aws-sdk-go中才有，上传和下载时，都通过签名实现，而不需要返回具体对象。

aws-sdk-go-v2中没有原生实现，可参考 https://github.com/aws/aws-sdk-go-v2/issues/438 自己实现

上传生成预签名

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// PutObjectRequest generates a "aws/request.Request" representing the
// client's request for the PutObject operation. The "output" return
// value will be populated with the request's response once the request completes
// successfully.
//
// Use "Send" method on the returned Request to send the API call to the service.
// the "output" return value is not valid until after Send returns without error.
//
// See PutObject for more information on using the PutObject
// API call, and error handling.
//
// This method is useful when you want to inject custom logic or configuration
// into the SDK's request lifecycle. Such as custom headers, or retry logic.
//
//
//    // Example sending a request using the PutObjectRequest method.
//    req, resp := client.PutObjectRequest(params)
//
//    err := req.Send()
//    if err == nil { // resp is now filled
//        fmt.Println(resp)
//    }
//
// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/PutObject
func (c *S3) PutObjectRequest(input *PutObjectInput) (req *request.Request, output *PutObjectOutput) {
    op := &request.Operation{
        Name:       opPutObject,
        HTTPMethod: "PUT",
        HTTPPath:   "/{Bucket}/{Key+}",
    }

    if input == nil {
        input = &PutObjectInput{}
    }

    output = &PutObjectOutput{}
    req = c.newRequest(op, input, output)
    return
}

github.com/aws/[email protected]/aws/request/request.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
// PresignRequest behaves just like presign, with the addition of returning a
// set of headers that were signed. The expire parameter is only used for
// presigned Amazon S3 API requests. All other AWS services will use a fixed
// expiration time of 15 minutes.
//
// It is invalid to create a presigned URL with a expire duration 0 or less. An
// error is returned if expire duration is 0 or less.
//
// Returns the URL string for the API operation with signature in the query string,
// and the HTTP headers that were included in the signature. These headers must
// be included in any HTTP request made with the presigned URL.
//
// To prevent hoisting any headers to the query string set NotHoist to true on
// this Request value prior to calling PresignRequest.
func (r *Request) PresignRequest(expire time.Duration) (string, http.Header, error) {
    r = r.copy()
    return getPresignedURL(r, expire)
}

签名解析出来，有三个返回值（下载同）

  • string：HTTP请求的url
  • http.Header：HTTP请求的Header
  • error：函数返回的报错

生成下载预签名

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
// GetObjectRequest generates a "aws/request.Request" representing the
// client's request for the GetObject operation. The "output" return
// value will be populated with the request's response once the request completes
// successfully.
//
// Use "Send" method on the returned Request to send the API call to the service.
// the "output" return value is not valid until after Send returns without error.
//
// See GetObject for more information on using the GetObject
// API call, and error handling.
//
// This method is useful when you want to inject custom logic or configuration
// into the SDK's request lifecycle. Such as custom headers, or retry logic.
//
//
//    // Example sending a request using the GetObjectRequest method.
//    req, resp := client.GetObjectRequest(params)
//
//    err := req.Send()
//    if err == nil { // resp is now filled
//        fmt.Println(resp)
//    }
//
// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/GetObject
func (c *S3) GetObjectRequest(input *GetObjectInput) (req *request.Request, output *GetObjectOutput) {
    op := &request.Operation{
        Name:       opGetObject,
        HTTPMethod: "GET",
        HTTPPath:   "/{Bucket}/{Key+}",
    }

    if input == nil {
        input = &GetObjectInput{}
    }

    output = &GetObjectOutput{}
    req = c.newRequest(op, input, output)
    return
}

分段获取

HTTP 分段获取资源（HTTP Partial Content）是一种机制，允许客户端在下载大型文件时，只请求文件的一部分，而不是整个文件。这对于网络传输较慢或带宽有限的情况下非常有用，因为它可以减少等待时间并降低带宽消耗。

当客户端发送一个 HTTP 请求时，可以在请求头中包含 Range 字段来指定所需的资源范围。该字段的格式为 bytes=start-end，其中 start 和 end 分别表示资源的起始字节和结束字节的索引。服务器在接收到带有 Range 请求头的请求后，会返回指定范围的资源内容，并在响应头中包含 Content-Range 字段来指示返回内容的范围，而且返回的HTTP Code是206。

以下是一个示例的 HTTP 请求和响应头的示例：

请求头

1
2
3
GET /large-file.txt HTTP/1.1
Host: example.com
Range: bytes=0-999

响应头

1
2
3
4
HTTP/1.1 206 Partial Content
Content-Type: text/plain
Content-Length: 1000
Content-Range: bytes 0-999/50000

在上述示例中，客户端请求了 large-file.txt 文件的前 1000 个字节。

  • Range: bytes=0-999，代表请求数据的第0个字节到第999个字节，一共1000个字节。

服务器返回了这部分内容，Status Code是206 Partial Content，并指示这个范围是文件的总范围中的一部分。

  • Content-Type: text/plain代表响应的内容是纯文本
  • Content-Length: 1000代表响应的文件大小是1000个字节
  • Content-Range: bytes 0-999/50000代表响应的是文件的第0个字节到第999个字节，50000代表这个文件总大小是50000字节。

通过使用 HTTP 分段获取资源，客户端可以逐步下载大型文件，按需获取所需的部分，从而提高下载效率和用户体验。这在视频流媒体、大型文件下载和断点续传等场景中非常常见。

真实HTTP请求如下

1
2
3
4
curl 'http://xxx' \
  -H 'Range: bytes=0-' \
  --compressed \
  --insecure

浏览器默认会使用分段获取，而且传入Range的截止字节大小为0，将文件大小交由服务端处理。

实现方案

一些播放器会自动在Header中携带信息，不需要手动传入分片大小。而后端服务则需要设置最大分片大小，避免出现前端传入分片过大，一次性获取所有文件，分片传输时区意义。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func ResourcePreview(w http.ResponseWriter, r *http.Request, pathParams map[string]string) {
    ranges := r.Header.Get("Range")
    token := r.URL.Query().Get("token")       // 一些播放器无法自定义header信息，需要将一些权限信息放在query中
    fileId := pathParams["file_id"]   				// 将文件id放在path中，可以实现url唯一，具备缓存条件

    download, err := logic.PreviewResource(r.Context(), token, fileId, ranges)
    if err != nil {
        w.WriteHeader(http.StatusInternalServerError)
        io.WriteString(w, err.Error())
        return
    }
    defer download.Body.Close()

  // 将请求的s3的header放入返回的header中
    for k, values := range download.Header {
        for _, v := range values {
            w.Header().Add(k, v)
        }
    }

    io.Copy(w, download.Body)

    return
}

logic层处理token，以及Range参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
initDownloadResult.Headers["Range"] = []string{"bytes=0-"}
    if ranges != "" {
        index := strings.Index(ranges, "bytes=")
        var start, end int
        if index != -1 {
            ss := strings.Split(ranges[index+6:], "-")
            if len(ss) > 0 {
                start, _ = strconv.Atoi(ss[0])
            }
            if len(ss) > 1 && ss[1] != "" {
                end, _ = strconv.Atoi(ss[1])
            }
        }
        if end == 0 || end-start > models.VideoDefaultContentLength {
            end = start + models.VideoDefaultContentLength - 1            //  使用默认分片大小
        }
        initDownloadResult.Headers["Range"] = []string{fmt.Sprintf("bytes=%d-%d", start, end)}
    }

s3上传和下载文件有两种方式，无论使用那种方式，都需要处理请求。

如果直接使用sdk的方式，上传时，需要将文件的Body io.Reader放入参数中。

下载时，将前端传入的Header放在结构体中，例如Range、一些缓存内容等

1
2
3
if ifNoneMatch, ok := ctx.Value("If-None-Match").(string); ok {
    input.IfNoneMatch = aws.String(ifNoneMatch)
}

将返回的Response放在返回的Header中，例如一些缓存内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
cacheControl := "max-age=86400,must-revalidate"  
if s3Object.CacheControl != nil {
    cacheControl = aws.StringValue(s3Object.CacheControl)
}
w.Header().Set("Cache-Control", cacheControl)                         	// 缓存控制方案
Etag := ""
if s3Object.ETag != nil {
    Etag = aws.StringValue(s3Object.ETag)
}
w.Header().Set("ETag", Etag)                                                            // Etag信息
lastModified := time.Now()
if s3Object.LastModified != nil {
    lastModified = *s3Object.LastModified
}
w.Header().Set("Last-Modified", lastModified.Format("Mon, 02 Jan 2006 15:04:05 GMT"))
expires := time.Now().Add(24 * time.Hour)
if s3Object.Expires != nil {
    expires = *s3Object.Expires
}
w.Header().Set("Expires", expires.Format("Mon, 02 Jan 2006 15:04:05 GMT"))

获取到的结构体中包含数据内容Body io.ReadCloser。

要实现分段获取，需要传入的Header参数：

  • Range，例如：Range: bytes=6815744-

需要将GetObjectOutput中放入Header：

  • Accept-Ranges，例如：Accept-Ranges: bytes
  • Content-Disposition，例如：Content-Disposition: inline; filename="xxx.mp4"
  • Content-Type，例如：Content-Type: video/mp4
  • Etag，例如：Etag: "40bd8bc04fb9cd7a5795121addcb11e6-35"
  • Content-Range，例如：bytes 6815744-11010047/535677112
  • Content-Length，例如：Content-Length: 4194304

实际请求

通过查看浏览器中的请求，可以看到播放视频时，播放器会发两个请求，第一个请求Header中不携带Range，用于获取视频信息，第二个请求才是获取视频具体信息，会附带Range信息

第一个请求

image-20230821171137242

第二个请求

image-20230821171904779

这里需要注意，如果是以HTTP请求预签名的方式，在服务端使用签名访问s3获取数据，然后返回给客户端。

一些语言中，在HTTP client端中会存在缓存，会将第一次请求的数据会缓存起来，导致第二个请求接受到的数据可能不满足Header中的大小，导致第二个请求的数据无法加载。

因此，需要将缓存中的数据读取出来，并且丢掉。（golang不需要）

1
2
3
if ranges == "" {
    response.Body.Close()
}