Moodle 2.0: File management and repository - file picker vs course files

I must admit, straight away, that writing this post has left me in tears. As I sob over my keyboard, my head is spinning in conflict about the way file management has been implemented in the brand new version of Moodle: Moodle 2.0. I'm hoping one day I'll be able to write a post called 'Dr. Strangemoodle. Or: How I learned to stop worrying and love the file picker.' That day is not today.

Why a new way of thinking?

One of the arguments that is being churned out by supporters of the new file system (lead Moodle people, I will refer to these as 'new paradigmites'), is that Moodle is not supposed to be a file repository/CMS (content management system – or even a basic equivalent), it is a learning management system (LMS) and we should not be encouraging simply uploading files. Now, I'm not going to criticise the inspirational pedagogic ethos here, but when you consider that providing some form of resource is what the vast majority of current teaching processes require, you would have thought that the file management system would be seen as the most important thing to get right in an LMS or CMS. In my view, we cannot assume all users are at the same online pedagogic level as the new paradigmites. Many tutors have not yet embraced simply uploading a PowerPoint revision aid or preparatory seminar reading.

The old

The 'old' model (which is referred to as 'legacy course files' to cement in our minds that this is the 'old way' of thinking), has a file store area for each course. This area mimics what users (teachers) are used to: it's like windows explorer, with folders and files. Links are then made to these files with 'resources' or even within resources such as HTML web pages which can have relative link paths. In the 'old' model, you upload a file and can refer to it from many resources. If you need to change that file, you overwrite it and it updates all resources.

The common problems with the old way were that you could not access these files from other courses and when you wished to copy an activity (eg a forum or quiz), you had to copy the entire course instead of just copying that activity in order not to lose attached files or quiz question files.

The new

The new model addresses this by attaching files to individual instances of activities or resource links. This also allows more rigorous access control, international filenames and access to everything regardless of where it is located (with the right permissions).

However, my biggest gripe is that we have now swung from one extreme to another. All that was possible previously is now awkward, and all that was impossible previously has been realised. I fear this may be down to a poor requirements analysis, where discussions focused on what was missing on previous versions and downplayed the essential requirements of current users.

Essentially, the Moodle 2.0 file system has been abstracted, so files no longer reside on the server in a logical 'tree' structure with the same filenames they would have on your office computer. This means that in order to decipher the files stored on your server's hard drive you need access to the database file table. Why this level of security through obscurity is required, on an already very secure space on the server without http access, I will never know. It is security through obscurity (read 'pointless') too, the files and content are still there, just renamed with random hash strings and with the file extension removed.
What is worse is that not only can you now not at a glance see which courses are hogging all your server space via the server, but that even within Moodle, in order to get to a simple list of files you have to be creating/editing a resource or be using the text editor! It also means that when using the standard Moodle 2.0 file management system it is impossible to use relative links within files (e.g. an uploaded HTML file pulling in images/css kept locally to the HTML file).

Duplication

At first, the new repository system was heralded as a way to access files from any part of Moodle. It seemed as if we could finally stop duplicating our files in every course. However, things are not as simple as that. When selecting files from an external repository with Add Resource > File, the Moodle 2.0 file picker in fact copies files from the original source (so every resource/activity has its own instance of the file), instead of providing links to them. Therefore, you have to ensure you create a link with Add Resource > URL.

When using Moodle's inbuilt repository things get slightly more complicated. If you upload a file to the Moodle server files, add it as a file in five courses, spot a mistake and upload a new version, the file does not change in all courses automatically but requires changing the link to the file for every instance. This is because the only way to change something is to delete and reupload (in order to keep other instances as they were). A lot of effort I think you'll agree. Similarly, imagine you upload a file which is then used by other tutors, you spot something dodgy in your original file and wish to remove it, making it offline. If other tutors have used this file, they have their own virtual copies. i.e. they're using version A on the server and you're now using version B, and you won't be able to track them down as deleting your original does not delete their copy. OK, so in a normal CMS or the old course files, there's nothing to stop the second tutor downloading and re-uploading a copy, but when you use the terminology of a 'repository' where you select a file, you are not expecting a new instance of that file to be created if you wish to change it.

ED: For a very clear explanation of the duplication problems caused be linking to files both within Moodle and repositories see this post by Elena Ivanova.

ED: Check the comments below which state that duplication is on the user point of view, and on the server it only occurs at point of re-uploading (or when using external repositories).

Comparison of file management between Moodle and Blackboard

In fact, far from resolving file management issues, the new Moodle 2.0 file 'paradigm' will cause more headaches. What has happened is that files are now associated even more deeply with elements of courses than previously. Although, yes you can access resources across the whole site, this is only leading to duplication as the system copies instead of links. What is required, as much as I really do wish to kill myself for saying it, is something like Blackboard's CMS system.

Linking files

  • Blackboard CMS model: Upload to a central repository (file system), link to the files.
  • Moodle 1.X model: Upload to a course repository (course files), link to the files.
  • Blackboard LMS model: Upload to an individual activity/resource.
  • Moodle 2.0 model: Upload to an individual activity/resource. Or, locate existing resource and Moodle will link to the file.

Storing and accessing files

  • Blackboard CMS model: Single, logical structured file system, username-based rights on individual files/folders. Access through CMS section of Blackboard (Content Tab).
  • Moodle 1.X model: Course-based logical structured file system, course-based rights on whole course file system. Access through file manager in each course.
  • Blackboard LMS model: Abstracted file storage, activity/resource-based rights. Access only through editing resource. Only able to access the items associated with the resource currently being edited.
  • Moodle 2.0 model: Abstracted file storage, activity/resource-based rights. Access only through going to edit an activity/resource, but can access other resources and other courses items through the file picker.

This may also best be represented with this little picture (note: consider Moodle 2.0 duplicated file to be a virtual duplication to the user, not on the server, until one of the files change):

Three models of CMS file management in VLEs (description below)

Above: Three models of file management on a VLE. Traditional CMS where you link from multiple courses to one copy of the file. Moodle 1.9 where a file resides in a course but can be linked multiple times in that course; manually copy if needed in another course. Moodle 2.0 where each and every resource/activity requires a new instance of that file, even within the same course; automatic duplication to another course on use.

But, this is how it is

Sadly though, I don't think the Moodle wider community will win this one. The core Moodlers seem insistent on the merits of this new file system, even though it is counter-intuitive, more costly in terms of server space, and difficult to navigate. Here's a quote from Martin Dougiamas, the lead Moodle developer, from this forum discussion: http://moodle.org/mod/forum/discuss.php?d=157250

"The answer is not to go back to the 1.x design - it's too late for that. If there are issues with the 2.x interface then let's work on them (as we have been for over 2 years). To me it's clear that the general trend in the world is to move towards specialised repositories that do that job really well, and then plug these into application servers like Moodle to use the files. I think most of us involved recognise that there is going to be some re-learning required, that old workflows may have to change, but the result is a more powerful, flexible and international tool with fewer problems."

I don't like to say this, and I do have a lot of respect for the core Moodlers, but as it's been brewing in my head all day I need some form of vent: There is always a danger when one spends time and effort on nurturing a new way of thinking, new code developed, that it leads to becoming blind to the needs of real users. A closeness to one's code can often lead to not stepping back and considering 'this is really not going to work for the average tutor out there.' I admit, that it's nothing great for me to come in at this point after a whole year's development on the system and essentially say 'you're wrong', but the impression that's being pushed from the Moodle forums is that it is us, old workflow type of people, are wrong in how we choose to use Moodle.

It could work

Theoretically, though I've not had a good look at the database structure and linking mechanisms, it shouldn't be too difficult to stop the duplication of files. Indeed, it looks like serious discussion is happening around providing options to either 'link' or 'copy'. If a file is known, a reference stored in the database, then all it requires is to tie that reference to instances of resources/activities, rather than the files. When backing up, that is the point at which you pull the file into the instance of the resource/activity. This database referencing also means that the file picker when browsing system files can instead use the same structure as course files, perhaps with two extra virtual folders for 'forum resources' or 'question resources' etc to help manage that side of things. After all, if there is abstraction in the true system, then the file picker is just a visual representation. Finally, the ability to look at the files (as in the file picker), rename, overwrite and delete from outside a resource/activity editor should be feasible too. It is highly unintuitive to have to go into and edit a resource in order to do something with an uploaded file, when it should be accessible directly.

Usability problems

One of the fundamentals to usability is to ensure the user doesn't have to remember. Unfortunately with the Moodle 2.0 in order to link to a file, you have to remember the course and activity the file was attached to in addition to the filename itself. Adding metadata or a search facility doesn't resolve this problem, as again it's dependent on the user remembering what meta taxonomy is being used, and attaching metadata in the first place (which the average tutor who just wants to get the job done isn't too fussed about).

The file picker: note the not-so lovely path to get to a file uploaded as the link 'test (file)' on the course page

Above: The file picker. Note the long, convoluted path to this Word doc, which was uploaded linked as 'test (File)' on the course page. There's a pointless 'Files and subfolders' virtual folder which is at least one extra click that could be got rid of. Second, why not simply bung this under 'Test Course A Full Name' as it is linked direct on the course page?

Overwriting files has also been sidelined. If you attempt to upload a file with a filename that already exists, you get notified, but aren't able to overwrite it. You have to either upload and save as a different file name (what the?!) or first delete the old file before re-uploading a new version. Since the days of DOS you've been prompted with 'File exists: Overwrite? Yes / No / Run away', so why another basic element of file management is missing here I do not know.

One of the suggested approaches is to set up a standard tree-like file structure repository folder. However, the only way to upload files to this repository is outside the Moodle interface, e.g. by FTP. This is not practical for any small-scale or institution-wide system where tutors may want to upload HTML mini-sites. Tutors don't have the right access, neither would we want to give them FTP access to the server!

The good stuff

What the new file repository approach has allowed though is the use of external resource sources, such as Google Docs, Flickr (Flickr Repository and Flickr Public) and Dropbox (Dropbox Repository). Integrating these external systems, which also encourage transferable IT skills (i.e. once someone leaves and institution they won't be using Moodle, but they may continue using various third-party giants like Google Docs), is definitely a step up in creating rich learning environments.

The private repository is also a great. Students have craved for a space to upload their own resources, and even better to be able to pull them directly into forum posts. I think this is definitely going to be one of the more positively received aspects of the new system.

Workarounds for HTML mini-sites and HTML-based learning resources in Moodle 2.0

Thankfully, the old way of uploading, accessing and linking to files still exists. However, it's not activated by default, is scorned upon by the new paradigmites. There are still (probably the majority) of structured courses using Moodle who have HTML-based content, woven together with CSS, images, documents and the like, uploaded in zips and unzipped in Moodle 1.9 preserving the relative paths linked in the HTML code. It took a lot of badgering to get a half-usable solution in the Moodle 2.0 release.

I've posted the solution on the Moodle User Forums and will create a proper guide on this site in due course.

Conclusion

So whilst my heart still belongs to Moodle, my head is having a rough time with the numerous usability flaws with content management. By the language (use of 'legacy' and 'still want') I think that we have a long way to go to address these issues from the user perspective, regardless of the aspirant pedagogical philosophy where Moodle becomes the centre of learning, not a centre of resources. There'll be more posts on Moodle 2.0, hopefully more positive!

Follow-up material

To really get your head around what is going on I strongly recommend reading these two posts by Mark Drechsler:

And, if you are a little lazy, just view his SlideShares:

The latest incarnation of the 'bug' tracker around the file system is here:

Some background discussions are on the Moodle forums here:

Comments

Thank you You have let me feel that I am not crazy

Matt, I just found your post from early this year regarding Moodle 2.0 and the new file management system .I wish to thank you for an excellent post. I am still relatively new to Moodle and I am a one person shop for Moodle administration for Khalifa University in the UAE. I have worked in five different lms systems and I love Moodle, but I am very conflicted about upgrading to 2.2. I find the file manager and the picker confusing. And if I do a large number of the faculty will and this will stop them from using their Moodle rooms.

From the few postings I have done and in attending the one Moodle Moot I have found a highly negative reception to questions and concerns about the new file system. I felt like I was slapped down and sent to the re-training if I did not understand and support the new system.

I am also very concerned that the developers have lost their connection with the users and are creating a Moodle for themselves rather than one for us. They are not here to dictate pedagogy that is the job of our administration, our deans and faculty. We as professional educators are looking at a system that is the best for us and our students.

Another frustrated user here

Some of what they have done makes sense but some of it is completely illogical and makes Moodle 2 and in particular the repositories feature pretty much useless.

Having used previous versions I was excited by the repositories features of 2.x, and the ability to store videos (often up to 250mb in size) in a separate repository such as S3, and as such avoid clogging up our main Moodle server which was already struggling to cope with the load.

If I understand right though the repositories feature has been made next to useless in this scenario as instead of just authenticating with and then calling the files from S3, Moodle actually copies them onto the Moodle server which ends up using up just as much disk space as it always did.

That's when it works of course, as more often than not it crashes with an error after you have been staring at the file pickers equivalent of an egg timer for an hour trying to pick a 250mb video file.

I like Moodle having used it for some years, but I am so frustrated that I am very close to evaluating alternatives at this point and I suspect I am not the only one.

It will only copy when using Add resource > File

Hi there. It should work ok without copying if you use Add resource > URL, and then link to the file rather than pulling it across to the Moodle server. However, I'm not sure how the authentication works on that.

Really helpful post

Thanks for summarizing the issues so fully, and for providing your helpful suggestions on using the new Moodle file structure.

Spot on

I agree totally with this post. I've had a very frustrating experience which I spent several hours just trying to upload and edit a 75mb scorm package.

I'm a developer of many years, but only scorming for a couple. I must say the new file system is very confusing and not intuitive at all. If all you want to do is upload one file, sure its great. I have no idea how I'm going to try get my clients to become familiar with this tool.

After googleing through many posts, I finally found several work arounds and workflows to try achieve simple tasks, like how to ftp large files and then access. It shouldn't be this hard.

And basic things like a preloader status on file uploads - Its not a lot of fun trying to upload 75MBs and having no idea if its hung or if its even working - as well as randomly getting errors in the file picker, which require a reopen. The spinning endless loop without %complete is useless.

This may be due to my lack of knowledge, but in a scorm package, I only want to update the css, or manifest file, but only one file can be chosen in the scorm module and can't upload duplicates. It really was so much simpler in 1.9x via ftp to make tweaks without having to reload the whole thing.

Another comment was on lack of security for 1.9x. This is an issue for some sure, but for others it isn't.

In summary - sure it has potential to do some cool things - but this should not be at the expense of current work-flows, ease of use, or reduced functionality. At least the 1.x use case scenarios - should have documented equivalents for 2.x, rather then a frustrating guessing game.

Matt A

Good overview, but a couple of minor points

Hi Matt,

Thanks for putting the time into a post like this - the more people talking this issue through with constructive arguments the more likely we are to end up with a model in 2.0 which limits the problems of the Bad Old Days but at the same time doesn't cut off its nose to spite its face.

I deal with many who definitely do not fall into the category of being 'new paradigmites' on a daily basis, so trying to find a model that keeps everyone happy is very much in my own interest as well. I hope some of the 'robust' discussions in the tracker and the goals listed at http://docs.moodle.org/en/Course_files#Roadmap_for_future_improvements will one day soon lead those who find this an important issue (and I know there are lots of them) to a happier place.

I did want to point out a couple of things in your post which don't quite ring true to me though.

Firstly, the problems you've listed with 'the old' are understated. You've missed the lack of security around having access to files managed at a course level, the inefficiencies of backing up every single file uploaded to a course regardless of whether its used or not, and of the breaks encountered when you manually delete a file from the file structure. These problems have been very real in 1.9, and I believe in WebCT, for a long time, and while I don't dispute that the model has swung from one extreme to the other, its worth putting in the full case for why the old way had problems. Most people I meet tend to be fine with these problems, which I suspect is because they've been dealing with them for so long they are considered part of the furniture.

Secondly, the duplication you're talking about - its worth understanding that this is duplication at a user experience level. By this, I mean that there is less duplication in the storage of files deep in the bowels of Moodle because Moodle is now only storing one copy of any identical files. Upload the same 200Mb video file in two separate courses? Moodle will only store one copy in its database. The kicker though is that from the user experience perspective you are most definitely dealing with two separate logical views of that file, so for all intents and purposes, the end user is working with duplicated files, and can't get around that (aside from using Legacy Course Files). Its only at the database level (which you as the end user never see) where you should find that a 2.0 site with loads of re-used content used across multiple courses should be smaller than a corresponding 1.9 site, which would have most definitely duplicated the file in its file system based storage model. If you can manage to successfully explain that concept to an academic who is struggling to even use a browser then you're a better man than I - but if you can then it at least helps explain another reason why it is the way it is.

Cheers,

Mark.

Hey Matt.

How about updating your blog post to correct the errors of fact Mark writes about.
Your comments (which seem to be incorrect) are still being quoted elsewhere. eg http://moodle.org/mod/forum/discuss.php?d=169791

-Derek

Hi. I've clarified my wording

Hi. I've clarified my wording in the 'Duplication' section of the post. The best post by far which allows us to understand what goes on is by Elena: http://moodle.org/mod/forum/discuss.php?d=168495#p739240

Apologies for any confusion.

Quick comment

Here's a view:
It's going to work OK with a file used once in a site.
For lots of other uses though, Moodle really needs a full on repository. Alfresco or something.
Manage your site-wide files with lots of occurrences this way, link to the repository and update once there.

Re Mark's comment about explaining this. Agreed.
-Derek

Thanks Mark for your reply,

Thanks Mark for your reply, and for the corrections to balance out my initial overload of a 'what have they done' panic-stricken post. As Gyamfi also comments, I think my biggest annoyance is the implementation. Instead of user rights-based access they've gone down the route of obscurification by resource/activity based access which makes me think the system is a bit overly contrived.

I also stand corrected on the duplication front. Since my post, Elena Ivanova has written a very useful forum post on the possible combinations of repositories, resource types and linking mechanisms: http://moodle.org/mod/forum/discuss.php?d=168495#p739240 

Matt

RE: In support of the old

I think the new file system is not user friendly. Moodle used to have a hide/show approach to items, so why not use similar approach to files (including access rights list and not allowing direct linking to files). At least the files would be there and codes utilized for url access but at least the underlying files would be there. Other cmses do this so i think the Moodle people probably got this wrong.

On the issue of duplicates, moodle just needed to have several source areas (categories, groups, user, courses, blogs) so users could pick from anywhere.

Overall, I am a bit disappointed.

Gyamfi

Moodle 2 is depressing

Thank for this. Moodle 2's bad press is looking well deserved.

I tend to give up on software that cannot document itself with clarity. For goodness sake Moodle, tell me how.

I've struggled uploading to Moodle 2 and found it insumountable. To be frank Moodle 1 was a pig to use and but at least I could find the bacon.

Good analysis!

I think the new file handling system is going to prove a real showstopper for the vast majorty of the several hundred teaching staff I support. I spend a lot of time encouraging staff to use Moodle in more pedagogically sophisticated ways than for distribution of files and learning resources. Unfortunately, this still accounts for the vast majority of use and if this aspect of Moodle is made too cumbersome to be manageable they will turn away from using Moodle altogether rather than use this as a stimulus to up their game.

Bookmark and Share