{"id":136,"date":"2008-03-20T11:41:36","date_gmt":"2008-03-20T10:41:36","guid":{"rendered":"http:\/\/www.jurecuhalev.com\/blog\/2008\/03\/20\/listing-all-keys-in-s3-bucket-using-python-boto\/"},"modified":"2008-03-21T11:17:38","modified_gmt":"2008-03-21T10:17:38","slug":"listing-all-keys-in-s3-bucket-using-python-boto","status":"publish","type":"post","link":"https:\/\/www.jurecuhalev.com\/blog\/listing-all-keys-in-s3-bucket-using-python-boto\/","title":{"rendered":"Listing all keys in S3 bucket using python boto"},"content":{"rendered":"<p><span class=\"zemanta-img\" style=\"margin: 1em; display: block; float: right\"><a href=\"http:\/\/www.flickr.com\/photos\/90684505@N00\/1645592932\/\" target=\"_blank\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/farm3.static.flickr.com\/2264\/1645592932_2123f3c875_m.jpg\" style=\"border: medium none ; display: block\" height=\"130\" width=\"196\" \/><\/a><span style=\"margin: 1em 0pt 0pt; display: block\">Source: <a href=\"http:\/\/www.flickr.com\/photos\/90684505@N00\/1645592932\/\" target=\"_blank\">Flickr<\/a><\/span><\/span>When using <a href=\"http:\/\/code.google.com\/p\/boto\/\" target=\"_blank\">python-boto<\/a> package to list keys in your <a href=\"http:\/\/en.wikipedia.org\/wiki\/Amazon_S3\" title=\"Amazon S3\" rel=\"wikipedia\" target=\"_blank\" class=\"zem_slink\">S3<\/a> bucket, you might hit a limit of 1000 keys when using function call <em>bucket.get_all_keys()<\/em>. In order to get full list of keys just do something along the lines of<br \/>\n<code><br \/>\nkeylist = [k for k in self.bucket]<br \/>\n<\/code><\/p>\n<p>since as it turns out, bucket has an <a href=\"http:\/\/en.wikipedia.org\/wiki\/Iterator\" title=\"Iterator\" rel=\"wikipedia\" target=\"_blank\" class=\"zem_slink\">iterator<\/a> over key name.<\/p>\n<p>(Discovered through <a href=\"http:\/\/cvs.savannah.nongnu.org\/viewvc\/duplicity\/duplicity\/backends.py?root=duplicity&amp;r1=1.42&amp;r2=1.43\" target=\"_blank\">patch of Mitchell Garnaat to Duplicity project<\/a>).<\/p>\n<p id=\"zemanta-pixie\" style=\"margin: 5px 0pt; width: 100%\"><a href=\"http:\/\/www.zemanta.com\/\" id=\"zemanta-pixie-a\" title=\"Zemified by Zemanta\"><img decoding=\"async\" src=\"http:\/\/img.zemanta.com\/pixie.png?x-id=7474d78a-e97d-4b0c-bdad-978483400539\" id=\"zemanta-pixie-img\" style=\"border: medium none ; float: right\" \/><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Source: FlickrWhen using python-boto package to list keys in your S3 bucket, you might hit a limit of 1000 keys when using function call bucket.get_all_keys(). In order to get full list of keys just do something along the lines of keylist = [k for k in self.bucket] since as it turns out, bucket has an [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[15,14],"tags":[150,147,152,148,151,932,149],"class_list":["post-136","post","type-post","status-publish","format-standard","hentry","category-python","category-tech","tag-amazon-s3","tag-boto","tag-bucket","tag-iterator","tag-mitchell-garnaat","tag-python","tag-s3"],"acf":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/posts\/136","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/comments?post=136"}],"version-history":[{"count":0,"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/posts\/136\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/media?parent=136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/categories?post=136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.jurecuhalev.com\/blog\/wp-json\/wp\/v2\/tags?post=136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}