I am really annoyed at how slow GitLab became in my last project, looking for something more scalable for the next one. Using LFS is not an option unfortunately. Does anyone have any recommendations for working with large repos on other tools (Bitbucket, CodeCommit, ...)?
Gitlab is getting too slow - any ideas?
If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.
These systems normally just store the difference to the previous state, which keeps the repositories small. But this only works with text files. When you change a binary file the whole file is stored, which bloats the repository and makes it slow.
I didn't solve this problem yet, I just don't add binary assets to the source repository at all. I have a Seafile instance running, which synchronizes my assets folder with a server.
13 hours ago, trill41 said:If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.
Most of them deal with large binary files pretty well... except for the DVCS systems like Git. Git doesn't cope with large files because every client stores the entire history, and because LFS (the hack to fix that issue) is build around an irreparably slow architecture on Windows.
My company has a single SVN repo containing every asset (mostly PNG and DAE files) and every built EXE file since 2012 -- about 20GB when the current version checked out on a client computer (plus 60GB for the .svn folder, WTF?!), but amazingly disk usage on the server is only about 26GB to store the full history because it apparently does manage to compress diffs of these files pretty well 
In my experience in the industry, Git, SVN and Perforce at the most common that I've seen.
@linus_e are you just looking for a Git alternative, or that plus a service provider that hosts repositories for you on the internet?
On 9/8/2019 at 11:55 AM, trill41 said:If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.
This is completely untrue for centralized VCSs like CVS, Subversion, or Perforce and only applies to DVCSs like git.
On 9/8/2019 at 11:55 AM, trill41 said:These systems normally just store the difference to the previous state, which keeps the repositories small. But this only works with text files. When you change a binary file the whole file is stored, which bloats the repository and makes it slow.
This is also untrue of most VCSs. Most VCSs (including git) store snapshots, not differences. While these are stored efficiently using compression and referencing unchanged files (svn) or blocks of content (git), they are still snapshots. The only exceptions I know are darcs and pijul.
The main problem with binary file diffs for size reduction (semantic diffs are a separate issue) and compression isn't even that they are binary, but that they are often already compressed (jpgs and pngs for example), so a small change might change almost everything.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
