Loading...
 
Features / Usability

Features / Usability


batch upload large (~100 GB) files

Hi,

I am using a wiki for archiving purposes and need to upload large files, up to several 100 GB. This should be possible through batch upload, which works fine up to 20 GB, on Tiki version 9.6. Larger files (~100 GB) result in an Internal server error message, suggesting a timeout somewhere. The file has actually been moved from the upload dir to the tiki dir used for file gallery storage with its name encrypted as usual, but this entry does not appear in the tiki_files table in the database. Questions:

1. what setting in php could affect this? php max_execution_time is 180, and storage quota on the server are ample.

2. tiki documentation for version 10 indicates that "It is now possible to integrate very large files into the Tiki File Gallery using Batch Upload. See Batch Upload for details." However, there is no definition of "very large files" there or in the Batch Upload referred to, so would an upgraded tiki version be able to handle `100 GB files correctly?

Thanks!
Joke

United States

You should probably review the upload_max_filesize
post_max_size settings, too. See the docs for details.

I've also seen many cases where the webhost imposes hard restrictions on upload size that cannot be overridden.

HTH,

- Rick | My Tiki Blog | My Tiki UserPage

Why be a dummy? Get smarty! TikiForSmarties.com
Tiki for Smarties, your source for the best (and only) Tiki books, guides, and tutorials.

Hi,

I increased all php variables relevant for upload limitations to large (250 GB) or unlimited when available (on a local install, so no server-side limitations). I still have problems with large files. Batch upload more or less works up to 20 GB, meaning that the files are stored in the target dir & included in the file gallery, but they display & download incorrectly when size > 2 GB. Either size displays as 2.0 GB (on live server installation), or as the actual but negative value (local test installation). In both cases, downloading the file will not work, downloads are 0 GB in all browsers (IE, Firefox, Chrome) I tested.
Actual file size = 2,265,686,016 bytes
Value in tiki file gallery after upload: -2 029 284 587 B
The uploaded file is ok, it can be downloaded again directly through ftp & unzips and reads correctly.

I noticed that newer tiki versions (10 and up) document "It is now possible to integrate very large files into the Tiki File Gallery using Batch Upload.
See Batch Upload for details." but it never defines how large "very large" is.

Has anyone been able to store large files, 10-200 GB, on tiki 10, 11 or 12 through batch upload yet?

Cheers, Joke


Upcoming Events

1)  Wed 23 Sep 2020 14:00 GMT-0000
September TRM

Why Register?

Register at tiki.org and you'll be able to use the account at any *.tiki.org site, thanks to the InterTiki feature. A valid email address is required to receive site notifications and occasional newsletters. You can opt out of these items at any time.