It's almost certainly going to be that the amount of memory required is not available. Using garbage collection, 'import gc', and calling 'gc.collect()' before and after issuing a 'requests.get()' may help but you won't be able to download anything larger than the memory available doing it that way.
You should be able to handle larger downloads by using TCP sockets. Open a connection to the site, issue a "GET" request, read the returned data in chunks, remove the headers, save the data to Flash or SD Card, or extract the data you are looking for and discard the rest. That will work for 'http' but I have no idea how to do it with 'https' and it can get a little complicated when having to deal with multi-part MIME.
Bottom line is that it's not easy dealing with large web pages. It's natural to try accessing sites like 'google.com' when testing things but that often doesn't work. Smaller pages should.
You should be able to handle larger downloads by using TCP sockets. Open a connection to the site, issue a "GET" request, read the returned data in chunks, remove the headers, save the data to Flash or SD Card, or extract the data you are looking for and discard the rest. That will work for 'http' but I have no idea how to do it with 'https' and it can get a little complicated when having to deal with multi-part MIME.
Bottom line is that it's not easy dealing with large web pages. It's natural to try accessing sites like 'google.com' when testing things but that often doesn't work. Smaller pages should.
Statistics: Posted by hippy — Wed Aug 07, 2024 11:34 am
