Home arrow FAQ arrow Remository general arrow File storage in Remository
File storage in Remository
Contributed by Administrator   
May 08, 2007 at 11:47 AM

Versions of Remository from 3.40 will store files in the database by default. The reasons for this are given below. For a moderate sized repository, database storage will work very well. Do not expect to see uploaded files in the file system, as they will be stored in the database table xxx_remository_blob. A moderate sized repository is one where most of the files are no more than one or two megabytes in size, and there are no more than 100 files.

If you plan a large repository, then it is worth spending time thinking about how you want to manage it. Please review the factors discussed below.

File storage is determined on a container by container basis. For a container to hold its files in the file system, it must have an absolute path recorded (an absolute path is a path that goes right back to the root; it is NOT a URI). Go to the admin interface of Remository, container management, and edit your containers. To make things easy, there is a flag for whether the default storage is database or file system. BUT this only affects new containers. Do not forget to check the sample container created by Remository, as it will initially be created to use the database. New containers, after the default is set to file system, will be given the default download path from the overall Remository configuration if no value is set while creating the container.

Why was database storage implemented for Remository?

There are a number of significant advantages to database storage. The file system is vulnerable to problems stemming from file access rights. These are made complex by various factors to do with the way hosting is usually operated and the impact of file permissions. For more detail on this topic, please read my paper on Permissions and Hosting. Because of these, it has been quite common for first attempts at installing Remository to fail, and it is difficult for novice webmasters to figure out the problem.

The database is a much more consistent environment, and placing files in the database gives a very high chance that Remository will work on first installation. In addition, the database is very secure, as the files do not exist in the file system and therefore cannot be stolen or used in an attack on the site. Name clashes are not an issue and files with "dangerous" extensions, such as .php, can be safely stored. Moving files is very efficient since the actual file is never moved, only the information about it.

Although there was a delay in starting download of very large files from the database in Remository 3.41, this is fixed in 3.42. Even extremely large files can be stored in the database quite efficiently. There will be about 16 database requests per megabyte on a download. I have tested files up to 130 MB in size and others may have gone further. Likewise, it is possible to build very large tables in MySQL.

However, it can be quite difficult to handle large database tables through standard hosting mechanisms. For example, phpMyAdmin will typically refuse to handle an import of more than 2 MB, which is quite restrictive. To work effectively on large database tables, you will need access to the MySQL backup tools and the ability to move very large files around the server.

It is not all bad, though, since managing a sizeable collection of large files is not a straightforward task either. The issues are not in Remository, simply in the nature of the task being handled. Dealing with many gigabytes of data is not an easy task, and moving files around is liable to be slow. So Remository gives you the choice to keep files in either the database or the file system. You can even make this choice differently for different containers, there is no need for the whole repository to use the same storage mechanism.

If you want advice on a specific application, this is available as a custom service. Please use "Contact Us" to discuss your needs.

Last Updated ( Jun 27, 2008 at 09:21 PM )
<Previous   Next>

Save FM Radio


Subscribe to Premium Support

Get priority support for Remository and Glossary, sign up now for a Premium Support monthly subscription:

Your Remository user name

Or purchase a year's support:

Your Remository user name

Who is Online

Remository welcomes guests and visitors

Real Time Web Analytics